CS Paper Radar

Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial

Xiaomeng Li

cs.HC cs.CVq-bio.QM ·submitted 2026-10-19 15:00 UTC · v2
code/data

Abstract

Photo-based estimation of food energy could ease dietary assessment, and general-purpose multimodal large language models (LLMs) make such estimates easy to obtain. Most evaluations use one photo taken before eating and compare with energy served, while dietary assessment needs energy eaten. We evaluated eight models from three vendors (Claude Haiku, Sonnet and Opus 5.5; GPT-6 luna, 6.1 sol and 6 astra; Gemini 3.8 Flash and 4 Argon) on 802 meals from ACETADA, a controlled feeding trial with weighed intake, with the before photo only, before and after photos in separate calls (subtraction), or both photos in one request. Three results ran against intuition. Subtraction under-estimated intake for six of eight models, because the models still report hundreds of kilocalories for a nearly empty tray. Scaling the before-only estimate by the average share eaten (0.87) matched much of the gain from the after photo; beyond that constant, the second photo lowered meal-level error by only 2.2 to 4.4 points, mainly in meals with food left. With one photo, two models had a near-zero intake bias only because they under-estimated the served energy and ignored uneaten food. In ablations, adding the after photo improved accuracy under both prompt formats for seven models, while prompt effects varied across models. Over 148 feeding days, five models kept the 90% confidence interval of their mean error within a pre-set 10% margin; one model also met the pre-set 30% limits of agreement, narrowly (-28% to +27%; limits outside 30% cannot be excluded), and the other seven did not. Protein was under-estimated by every model, most in protein-dense meals; repeated calls changed error by at most 1.4 points; a high reasoning effort helped only the smallest model. A low average bias is not evidence of accurate individual intake estimation; such estimates should support, not replace, assessment by a dietitian.

Comment
20 pages, 1 figure, 12 tables. v2: adds Gemini 3.8 Flash and Gemini 4 Argon under a pre-registered addendum; all tables recomputed over eight models; conclusions unchanged except that one model now meets the pre-set limits of agreement, narrowly. Code, prompts, per-meal model outputs: https://github.com/xtea/llm-intake-validation

Closest papers

Nearest neighbours by meaning across the whole index.

Single-Image Food Mass Estimators Do Not Transfer Their Gram Scale Across Datasets

cs.CV ·2026-10-21 ·Yongsheng Lian
benchmarkcode/data similarity 0.71
Abstract · 19 pages, 6 figures, 13 tables. Code, set lists and model weights: https://github.com/yongshenglian/foodmass-transfer

Estimating food mass from a photograph is central to image-based dietary assessment, but whether trained models transfer to new cameras and foods is unclear. We fine-tuned DINOv2-base on 2,253 weighed Nutrition5k dishes to predict mass from one RGB photograph and compared it with a zero-shot vision-language model, Qwen3-VL-32B, on Nutrition5k and three external datasets: ASA24 studio portions, ECUSTFD whole foods and Food Portion Benchmark meal trays. In domain, the fine-tuned model's mean absolute error was 24.8 g, less than half of Qwen's. On external datasets it did not beat Qwen. For a food it had never seen, it ranked portions by size, but no better than a count of food pixels. Its gram scale was wrong in two ways: it changed with how much of the scene the photograph showed, and it was off by a different factor for each food. Training on other foods or trays from the tested dataset cut the external error by 38 to 65%. DINOv2-large, a larger backbone trained on the same dishes, did not reduce the external error. One labeled photograph of the same food cut the model's ASA24 error to 28.3 g, better than that photograph achieved alone. Prediction intervals that held the true mass for 90% of Nutrition5k dishes held it for at most half of external photographs. Nutrition5k often photographed the same plate several times, so a random test split shared near-identical photographs with training and overstated accuracy. Single-image mass estimators therefore need labeled photographs from the setting where they are used.

0.34

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

cs.CV cs.AIeess.IV ·2026-10-08 ·Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
similarity 0.70
Abstract · 5 pages, 2 figures, 3 tables. Submitted to IEEE ICASSP 2027

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.

0.15

NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models

cs.CV cs.AI ·2026-11-05 ·Lubnaa Abdur Rahman, Ioannis Papathanail, Maria Panagiotou, Nefeli Dimitropoulou, Margret Harðardóttir +3 more
similarity 0.70
Abstract

Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.

0.05

Toward Low-Burden Image-Based Dietary Assessment Through Human-AI Collaborative Correction

cs.HC ·2026-10-30 ·Darren Liu, Andrew Lu, Zhiyuan "Ryan" Wang, Hanqi Luo, Xiao Hu +1 more
similarity 0.68
Abstract · 28 pages, 5 figures, 5 tables. Includes appendices

Image-based dietary assessment can automate initial food logging, but meal photographs often leave food identity, hidden ingredients, preparation, and portion uncertain, making correction a recurring part of use. We introduce Human-AI Collaborative Correction, which reframes correction as communicate intent $\rightarrow$ regenerate $\rightarrow$ verify: users express what should change through text, voice, or guided clarification, the AI regenerates the linked dietary record, and users verify the result. In a within-participant study with 30 participants, compared with conventional Manual Search-and-Edit Correction, collaborative correction was associated with lower workload (Raw TLX: 35.83 vs. 55.00; p < .001), 21.0% lower recorded usage time, 51.1% fewer recorded clicks, and more favorable user-experience ratings; 90.0% preferred it overall. An exploratory nutrient analysis also found lower energy MAE. These findings support intent-driven regeneration as a promising interaction pattern for reducing correction burden while preserving user verification and making image-based dietary tracking easier to sustain.

0.20

Predicting Postprandial Glycemic Response from Meal Images, Clinical Variables, and Gut Microbiome Information

cs.AI ·2026-09-21 ·Varvara Kondratyeva, Kamilia Zaripova, Nassir Navab, Azade Farshad
similarity 0.61
Abstract · 11 pages (9 text + 2 references). This is a paper first submitted to MICCAI 2026 MultiTab workshop prior to peer review. The final revised v…

Predicting postprandial glycemic response (PPGR) is fundamental to personalized nutrition and type 2 diabetes management, yet existing approaches typically rely on manually reported dietary intake, limiting their scalability in free-living settings. We propose a multimodal framework that replaces manual dietary logging with image-derived macronutrient estimates and integrates them with clinical variables and gut microbiome information for personalized PPGR prediction. The framework jointly performs image-based macronutrient estimation and glucose prediction, while an attention-based prediction module models interactions between dietary and host-specific information. We evaluate the proposed approach on a real-world dataset comprising meal images, continuous glucose monitoring, clinical variables, and gut microbiome profiles. The proposed model outperforms existing PPGR baselines using image-derived nutritional inputs and approaches the performance of methods that rely on manually reported macronutrients despite using automatically estimated nutritional information. These results demonstrate that combining image-derived nutrition with complementary clinical and gut microbiome information provides a practical foundation for scalable personalized PPGR prediction.

0.21

Investigating the Ability of Large Language Models to Analyze Recipes for Diabetes

cs.CL cs.AI ·2026-09-03 ·Revathy Venkataramanan, Aditya Luthra, Venkatesan Nadimuthu, Amit Sheth
benchmarkdataset similarity 0.57
Abstract

Several studies have evaluated the ability of Large Language Models (LLMs) for meal planning, yielding positive outcomes. These models can process natural language inputs and leverage learned knowledge from their pretraining to generate meal plans. In this work, we investigate the ability of LLMs to analyze the suitability of given recipes for diabetes. The primary challenge for LLMs is to retrieve relevant dietary guidelines for diabetes, decompose recipes into ingredients and cooking methods, and apply these guidelines to determine the recipe's suitability. To study these challenges, we employ three kinds of prompts namely, (i) Direct Query Prompt (ii) Context-Guided Prompt, and (iii) Exemplary Context Prompt that incorporate different levels of diabetes dietary guidelines from medical sources. We introduce a benchmark dataset curated for this investigation consisting of 7607 recipes that include 3807 recipes suitable for diabetes and 3800 recipes not suitable for diabetes. Our results demonstrate that most LLMs are cautious in predicting recipes as suitable to prevent detrimental outcomes. Further, the models that can reason using the dietary guidelines performed better in predicting the suitability of recipes for diabetes. Overall, Mistral-7B and Llama 70B showed superior performance to their counterparts.

0.14

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

cs.CV cs.AI ·2026-09-27 ·Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol +2 more
similarity 0.57
Abstract

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

0.17

Supporting Habit Formation in Pre-Meal Photo Capture: A Longitudinal Study of Behavior Change Techniques

cs.HC ·2026-11-02 ·Jun Fang, Yueying Suo, Yuntao Wang, Susu Wang, Xiaoyu Yang +3 more
similarity 0.57
Abstract

Pre-meal photo capture is lightweight only with a smartphone, yet habit formation requires repeatedly initiating and completing the action within the narrow window before the first bite. We examine how supports based on behavior change techniques can shape capture adherence, perceived habit, and support needs over time. Using a participant-informed selection process, we implemented three smartphone supports targeting Prompts/cues, Focus on past success, and Social support. We evaluated them in a five-week field study with 54 participants, spanning support, withdrawal, and reinstatement, followed by interviews and a three-week follow-up. Across 4,945 main meals, 63.9\% were confirmed pre-meal captures, and adherence was broadly maintained across phases. Perceived automaticity and spontaneous remembering improved mainly during initial support. As capture became more familiar, participants combined spontaneous remembering with deliberate strategies, reduced active feature use while continuing to value support availability, and carried the practice forward through accumulated records, familiar tools, and chosen relationships. These findings frame BCT support for pre-meal photo capture as a configurable arrangement. Systems should preserve a predictable capture pathway while adapting when, how, and through whom support remains available.

0.18