CS Paper Radar

Hear What You Eat: In-Ear Passive Acoustic Sensing for Multi-task Dietary Monitoring

Feiyu Han, Mingtian Zhu, Dawei Yan, Weiwei Jiang, Yuanhao Feng, Yanfei Zhang, Panlong Yang

cs.HC ·submitted 2026-11-11 01:06 UTC · v1

Abstract

The increasing obesity health concerns are driving the advancement of continuous dietary monitoring. Yet, existing single-sensor solutions still face challenges of coarse-grained sensing resolution, while multi-sensor fusion strategies encounter significant engineering complexity or hardware requirements. In our work, we present a single modality-based multi-dimensional dietary monitoring and logging solution, named EARDIET, which leverages inward-facing microphones on earables to capture in-ear chewing sounds for fine-grained diet analysis. With the texture audio profiles derivation, the pipeline consisting of signal processing and lightweight deep learning, is carefully designed to achieve eating episode detection, food texture sensing, and food type recognition simultaneously. Comprehensive experiments involving 32 participants with diverse BMI profiles are conducted to validate the effectiveness of multi-dimensional dietary monitoring. Compared with the state-of-the-art solutions, EARDIET can derive multi-dimensional dietary information with only a single modality, outperforming by about 10% in terms of eating episode detection. Furthermore, through real-world clinical validation, our solution can achieve masticatory disorder identification with an averaged F1-score of 85.97%, demonstrating potential for healthcare in the future.

Closest papers

Nearest neighbours by meaning across the whole index.

Oto-Meal: Earable Sensing with PPG and IMU for Personalized Meal Awareness

cs.HC ·2026-08-17 ·Yuxuan Hou, Jiao Li, Linshan Jiang, Jin Zhang
published similarity 0.69
Abstract · Accepted to WellComp 2026 (Workshop at UbiComp/ISWC 2026)

Meal awareness can help people reflect on hydration, chewing rhythm, and conversation-heavy meals, but many eating-sensing approaches rely on cameras, microphones, food photographs, or repeated self-logging. PPG and IMU offer a narrower sensing path by capturing physiological and motion patterns around meal-adjacent actions without raw audio, video, or photographs. We present Oto-Meal, an audio- and image-free earable prototype. Its pooled neural recognizer uses a two-stage event/rest gate and five-class behavior classifier. Separately, a within-user protocol evaluates a lightweight memory matcher built from labeled target-user examples. We invited seven volunteers and collected a seven-user dataset for mixed-user training, within-user memory evaluation, and modality ablation. The pooled model reaches 70.99% event accuracy. Under the separate memory protocol, 20% target-user calibration reaches 80.38 $\pm$ 0.84% event accuracy and 81.77 $\pm$ 0.69% cascade accuracy; with 60% calibration, PPG+IMU reaches 85.13 $\pm$ 0.57% event accuracy and outperforms IMU-only and PPG-only. These preliminary results suggest that audio- and image-free earable sensing with inspectable personalization can support low-burden meal-awareness review.

0.31

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

cs.CV cs.AI ·2026-09-27 ·Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol +2 more
similarity 0.60
Abstract

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

0.17

RF-Based Macronutrient Estimation in Liquid Foods via Composition-Aware Multi-Task Learning

cs.NI ·2026-11-14 ·Ishtiaque Ahmed Showmik, Kannan Athreya
similarity 0.58
Abstract

Current approaches to macronutrient estimation range from high-overhead laboratory analysis to user-dependent methods such as manual nutritional lookup, reading package labels, and vision-based image analysis. These approaches are often time-consuming, input-dependent, or sensitive to imaging and measurement conditions. In this work, we investigate synchronous 2.4 GHz OFDM radio-frequency (RF) measurements for estimating carbohydrate, fat, and protein content in liquid foods. Using a controlled sensing setup, we study how liquid composition shapes frequency-domain RF responses -- a first step toward a sensor integrated into a liquid-contacting vessel that estimates macronutrient content passively and label-free, and toward composition-aware RF food analysis beyond liquids. We propose the Ingredient-Conditioned OFDM Transformer (ICOT), a structure-aware multi-task Transformer that operates on two-dimensional OFDM channel frequency response matrices. ICOT infers canonical ingredient categories representing dielectric-driver composition and uses them to condition macronutrient regression through feature concatenation. On 28,420 captures spanning 55 beverage classes, ICOT achieves an in-distribution MAE of 0.101 g/100 ml and an out-of-distribution MAE of 0.485 g/100 ml -- comparable in magnitude to deviations present in manufacturer-reported labels -- though generalization to novel formulations remains the primary open challenge.

0.31

Reference-Free Food Volume Estimation from Egocentric Stereo Video

cs.CV ·2026-10-08 ·Vivan Kumar, Gautham Vinod, Fengqing Zhu
similarity 0.56
Abstract · 9 pages, 4 figures, 1 table

Current stereo vision-based approaches for food portion estimation are hindered by the user burden of using a mobile phone to capture multiple images or placing a reference object in the scene for scale calibration. As a step towards passive dietary intake monitoring, we introduce a method that uses stereo video from a wearable egocentric device to reconstruct food items in true physical scale to accurately estimate volume. By utilizing FoundationStereo for zero-shot stereo matching, SAM 2 for food segmentation, and TSDF fusion over a set of keyframes, our method yields a visually and metrically accurate 3D mesh of any plated food item from a single stereo recording. We leverage the unique offerings of wearables, such as a factory-calibrated stereo baseline and camera poses estimated on-device via visual-inertial odometry (VIO), to perform fully automatic reconstruction without a scale reference object in the scene. We show strong initial volume estimation results across five recorded food items, achieving a mean absolute percentage error of 9.1%.

0.22

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

cs.CV cs.AIeess.IV ·2026-10-08 ·Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
similarity 0.56
Abstract · 5 pages, 2 figures, 3 tables. Submitted to IEEE ICASSP 2027

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.

0.15

Diffusion-Based Synthetic Data Pretraining for Enhancing Activity Recognition

cs.LG cs.AI ·2026-10-01 ·E. Riveros, D. Vega-Oliveros, A. Soriano-Vargas, A. Rocha
similarity 0.55
Abstract · 6 pages, 3 figures, 1 table

Human activity recognition (HAR) is increasingly important for healthcare, well-being, and daily monitoring ap- plications, for which detecting alimentary activities such as eating and drinking can provide actionable insight into dietary habits and chronic disease management. HAR systems, however, often underperform on subtle and underrepresented classes, limiting their utility in real-world dietary monitoring. This work builds upon CABiGRU, a convolutional architecture with Bidirectional GRU layers, multi-head attention, and residual connections, designed to capture discriminative temporal patterns from smart- watch accelerometer, gyroscope, and magnetometer data. To improve CaBiGRU's generalization and reduce underfitting in the minority class, we leverage synthetic sensor data windows using a diffusion model and adopt a two-stage training strategy: pre-training CABiGRU on synthetic data, followed by fine-tuning on the real-world data. On the DEO (drinking/eating/other) dataset, the proposed pipeline achieves a balanced accuracy of 90.6%, improving over a strong supervised baseline and showing the benefits of diffusion-based synthetic pre-training for recognizing alimentary activities and representing a step forward dealing with unbalanced classes. These results suggest that combining diffusion-generated data with targeted fine-tuning enhances robust recognition of dietary behaviors, supporting more reliable deployment in healthcare and nutrition-monitoring settings.

0.18

Understanding Behavioral Dark Patterns of High BMI Individuals

cs.HC ·2026-08-29 ·Manjeet Yadav, Prasenjit Karmakar, Suchetana Chakraborty
published similarity 0.54
Abstract · 5 pages, 5 figures, accepted to DiversityOne Open Challenge at UbiComp/ISWC 2026

Understanding how everyday behaviors influence body weight is essential for designing effective and personalized health interventions. Existing studies largely rely on self-reported questionnaires or limited sensing modalities, making it difficult to capture the temporal dynamics of daily behavior. In this work, we analyze the DiversityOne dataset, comprising four weeks of passive smartphone sensing and ecological momentary assessments collected from 453 university students across eight countries. We extract behavioral features spanning dietary habits, physical activity, screen time, and smartphone usage, and investigate their associations with self-reported Body Mass Index (BMI). Beyond feature-level analysis, we employ Hidden Markov Models (HMMs) to uncover latent behavioral patterns. Our analysis reveals that higher BMI is associated with more frequent consumption of soda, alcohol, and processed meat. We further reveal that overweight and obese individuals spend longer periods in food delivery apps and are more likely to transition back to unhealthy eating and drinking routines after starting to exercise. In contrast, normal-weight individuals lead a more balanced lifestyle. These findings highlight key behavioral patterns that make weight loss particularly challenging.

0.42

MAST: Label-Efficient, Robust, and Generalizable Sound Detection for Biodiversity Monitoring via Masked Audio Pretraining and Self-Training

cs.SD cs.LG ·2026-09-14 ·Tianyi Xu, Daniel Pimentel-Alarcón, Zuzana Buřivalová, Claudia Solís-Lemus
similarity 0.52
Abstract · 29 pages, 8 figures

Passive acoustic monitoring can measure biodiversity at larger scales, but time--frequency annotation of animal vocalizations is expensive, site-specific, and difficult to sustain at scale. We present a label-efficient sound detection framework that combines masked audio pretraining with a lightweight detector on mel spectrograms, then further improves robustness through iterative self-training on unlabeled audio. We first pretrain a ViT-based encoder on unlabeled recordings via masked reconstruction and transfer the encoder to a detection backbone. To better separate animal sounds from confounding background, we add a box-level contrastive loss that pulls matched event regions together while pushing noisy negatives apart. We then apply a two-stage pseudo-labeling curriculum to exploit large unlabeled pools without additional annotation. We evaluate the performance on two ecologically distinct domains: tropical rainforest soundscapes (Indonesia) and bird vocalizations in Mediterranean habitats (Spain). On both domains, masked audio pretraining and contrastive learning consistently improve time--frequency detection under temporal and cross-site distribution shift, and self-training yields further gains in out-of-distribution performance. On the rainforest domain, MAST with self-training achieves +0.22 mAP and +0.24 F1 over the strongest baseline under cross-site shift. On the bird domain, self-training achieves +0.12 mAP and +0.10 F1 over the strongest baseline under cross-site shift. Overall, our results show that MAST can effectively extend self-supervised audio representations from clip-level tasks to robust box-level localization across diverse bioacoustic settings, providing a practical path for biodiversity monitoring with limited labels.

0.51