CS Paper Radar

NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management

Yulin Wei, Xiangchen Wang, Jianhui Pan, Jinyu Xiao, Zheng Tan, Ruozai Tian, Guanhua Chen, Feng Zheng

cs.CV ·submitted 2026-09-07 07:34 UTC · v1
top venuebenchmarkcode/data

Abstract

An embodied kitchen assistant must do more than recognize food in isolated frames. It must track ingredient states over time and integrate visual observations with recipe and nutritional knowledge to support constraint-aware decision-making. We formalize this capability as \emph{Embodied Nutrition Management}: perceiving nutrition-relevant events, maintaining a persistent food state, and using it for knowledge-grounded planning. Existing benchmarks evaluate static food understanding or embodied cooking actions, but do not measure whether an agent can continuously update and use nutrition-relevant states in dynamic kitchens. To fill this gap, we introduce \textbf{NutriBench-Kitchen}, a benchmark containing 1,500 manually verified question--answer pairs from 160 cooking videos. It covers five task families: Ingredient Entry, Memory Management, Recipe Query, Long-Term Planning, and Short-Term Planning, spanning food-state construction, maintenance, knowledge retrieval, and decision-making across different planning horizons. Evaluations of proprietary and open-source large vision-language models reveal a substantial gap from human performance, particularly in quantitative ingredient estimation, long-term state tracking, and reasoning under interacting constraints. We further introduce \textbf{Nutri-Vgent}, a diagnostic long-video agent with separate episodic, food-state, and recipe memories. Its consistent improvements demonstrate the value of explicit state representations and structured memory for nutrition management. Together, NutriBench-Kitchen and Nutri-Vgent provide a testbed for studying persistent state tracking and knowledge-grounded reasoning in dynamic kitchens. Code is available at https://github.com/V1ol1n/NutriBench-Kitchen.

Comment
17 pages, 4 figures, ECCV

Closest papers

Nearest neighbours by meaning across the whole index.

Nutri-ATLAS: Embodied Agent for Tabulated Lookup and Assistance for Smarter nutrition

cs.AI ·2026-09-26 ·Uttej Kallakuri, Boxun Hu, Ankur A. Butala, Najim Dehak, Tinoosh Mohsenin
similarity 0.79
Abstract

Generative and Agentic IoT systems offer a promising foundation for digital healthcare applications that combine sensing, personalized reasoning, and autonomous interaction in real-world environments. Nutrition assistance is a natural use case, but existing Large Language Model (LLM)-based systems are often limited to passive text interaction and static context, making them unreliable when food descriptions are ambiguous or nutritional evidence is missing. We propose Nutri-ATLAS, an Embodied Agent for Tabulated Lookup and Assistance for smarter nutrition in the real world. It integrates graph-grounded nutrition reasoning, hardware-aware LLM selection, and robot-based evidence acquisition. Nutri-ATLAS builds a unified Food-Nutrient knowledge graph from USDA FoodData Central and FoodKG and learns 64-dimensional GATv2 food and recipe embeddings. A shared hybrid graph-text scoring mechanism supports food nutrition extraction, nutritional gap filling, substitute retrieval, and recipe-level meal composition, while an LLM-guided skill interface navigates landmarks, updates dietary-context and food-accessibility memory, and grounds recommendations in observed food availability. We evaluate Nutri-ATLAS across nutrient estimation, substitution retrieval, recipe recommendation, patient-profile adherence, edge deployment, and real-world embodied execution. On HealthyFoodSubs, the hybrid retriever achieves 37.9% MAP, 80.7% RR@5, and 90.1% RR@10. On NutriBench v2, Dense+GAT retrieval grounds nutrient estimation across nine quantized Qwen3.5-9B configurations. On PFoodReQ, Nutri-ATLAS reaches 78.8% MAP, 83.0% MAR, and 77.5% F1. A patient-profile study shows adherence to allergy and healthy-target constraints for all selected cases.

0.11

NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models

cs.CV cs.AI ·2026-11-05 ·Lubnaa Abdur Rahman, Ioannis Papathanail, Maria Panagiotou, Nefeli Dimitropoulou, Margret Harðardóttir +3 more
similarity 0.69
Abstract

Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.

0.05

Profile-Aware Trustworthy Recipe Generation with Planner-Critic Agentic Remediation

eess.SY cs.ET ·2026-10-02 ·Shanhong Liu, Pai Chet Ng, Konstantinos N. Plataniotis
benchmarkcode/data similarity 0.68
Abstract

Recipe generation from food images has practical value for intelligent cooking assistance, but traditional one-pass generation often overlooks user-specific safety requirements such as allergies, dietary restrictions, and preparation constraints. We propose PCAR, a Planner-Critic Agentic Remediation framework for trustworthy recipe generation. PCAR separates recipe planning from safety verification: a Planner Agent extracts ingredients and generates recipe drafts conditioned on the user profile, while a Safety Critic Agent audits each draft and provides structured feedback for remediation when violations are detected. This remediation loop enables unsafe recipes to be revised rather than directly returned or discarded. We evaluate PCAR on real food images with 100 benchmark user profiles across four backbone models, including proprietary and locally served open-source models. Results show that PCAR achieves strong safety and generation performance with capable backbone models, while preserving practical recipe quality.

0.30

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

cs.CV cs.AI ·2026-09-27 ·Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol +2 more
similarity 0.68
Abstract

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

0.17

Towards Generalizable Visually Grounded Exploration of Household Devices

cs.AI ·2026-09-01 ·Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che +2 more
top venuebenchmark similarity 0.68
Abstract · Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026

Recent advancements in Vision-Language Models (VLMs) have demonstrated impressive capabilities in static visual recognition and high-level semantic reasoning. However, current embodied exploration paradigms still heavily rely on imitation learning from human-annotated trajectories, which severely limits agents' generalization ability. The key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration: the ability to operate novel devices without manuals or specific training by actively grounding abstract world knowledge into fine-grained visual affordances. Yet, existing benchmarks fail to evaluate this capability: they generally rely on explicit documents and annotated trajectories, neglecting the dynamic Hypothesis-Interaction-Refinement process essential for functional device operation. To bridge this gap, we introduce VGEBench, a comprehensive benchmark designed to evaluate the generalizable visually grounded exploration capabilities of VLMs. Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction. Experimental results demonstrate that existing VLMs face significant challenges in translating semantic knowledge into physical execution and maintaining long-horizon state tracking.

0.36

Generative AI enables human-centered design of sustainable plant-based foods

cs.CE ·2026-10-11 ·Vahidullah Tac, Ellen Kuhl
benchmark similarity 0.67
Abstract · 25 pages, 4 figures, 7 supplementary figures

Dietary transitions away from animal-based meat could reduce food-related environmental burdens, but plant-based alternatives must compete on flavor, texture, and overall liking. Here we extend a generative AI framework with 469 plant-based burger recipes and use it to create six formulations across five primary ingredients: bean, chickpea, lentil, mushroom, and oat. An executive chef translates the AI-generated formulations into standardized preparation protocols and refines the ingredient quantities based on culinary expertise. The integration of generative modeling and culinary expertise defines a human-centered generative AI framework for food design. We evaluate the six burgers in a blinded sensory survey with 101 participants in an operating restaurant and quantify their land use, greenhouse gas emissions, eutrophication potential, and scarcity-weighted water use. We use Big Mac scores from a previous survey as an iconic commercial benchmark. The mushroom burger achieves mean flavor, texture, and overall liking scores similar to the Big Mac, with less than one-third of its estimated environmental impact. The other formulations reduce the estimated impact further, in several cases to near one-tenth of the Big Mac value, but receive lower liking scores. Mixed-effects models quantify associations between sensory attributes and human liking and retain predictive accuracy on independent survey data, with mean absolute errors no greater than 0.52 points on a seven-point scale. Together, these results demonstrate how generative design, culinary expertise, human sensory response, and environmental assessment can converge to create sustainable foods that people want to eat.

0.24

A framework for recipe data structure with applications for culinary and nutritional insights

cs.CL ·2026-08-12 ·Mansi Goel, Sumit Bhagat, Saloni Srivastava, Malav Patel, Shlok Vinodkumar Mehroliya +1 more
similarity 0.66
Abstract · Main Text (11 pages, 4 figures, 3 tables); Supplementary Information (6 pages, 5 figures, 1 table)

Cooking is a complex process that transforms raw ingredients into delicious and nutritious dishes, yet the recipes that encode this process remain largely free text; readable by people but not directly computable. Existing recipe collections capture fragments of this information, but no shared representation links a recipe's structured ingredient composition, its geo-cultural provenance, and its nutritional profile within a single queryable schema. We address this representation gap by formalizing a framework for recipe data structure that decomposes each recipe into typed ingredient entities, grounds those entities in a reference nutritional database, and annotates them with geo-cultural and dietary context. We present RecipeDB2, a structured compilation of 128,942 recipes with 35,474 ingredients from 32 regions and 99 countries. Ingredient phrases are parsed into seven culinary attributes using a transformer-based named-entity model; ingredients are linked to the USDA reference tables through a BERT embedding strategy (F1 = 87.90 on a manually adjudicated set of the 200 most frequent ingredients), yielding 148 nutritional parameters per mapped ingredient; a Random Forest classifier propagates 34 ingredient categories across the full vocabulary; and a deterministic, conservative rule set assigns each recipe a dietary style. Through RecipeDB2 (https://cosylab.iiitd.edu.in/recipedb2/), we demonstrate a scalable framework for making recipes computable, turning culinary heritage (long treated as an artistic rather than a quantitative object) into a data-driven analysis.

0.22

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

cs.CV cs.AIeess.IV ·2026-10-08 ·Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
similarity 0.66
Abstract · 5 pages, 2 figures, 3 tables. Submitted to IEEE ICASSP 2027

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.

0.15