Calorie Lens

A photo of a meal goes in; out come the ingredients on the plate, how many grams of each, and the nutrition of every ingredient and of the whole plate (kcal, protein, fat, carbs, fiber, sugars, saturated fat, sodium). It is two models fused into one network, trained on the Nutrition5k dishes, whose every ingredient was weighed on a scale, and on USDA FoodData Central.

Model

photo ─► backbone (ConvNeXt-B or DINOv2-B) ─► 253 ingredient queries (cross-attention)
                                              ├─ is ingredient k on the plate?   (presence)
                                              └─ how many grams, if it is?       (grams)
         backbone ─► totals head ─► plate mass
grams_k = P(k) x grams_k, rescaled to the plate mass  ──►  253 neurons  =  stage 1's output
253 neurons ─► NutritionLayer (frozen, 253 x 8 per-gram table) ─► nutrition per ingredient + totals
food name ─► NutriText (own BPE tokenizer + 4-layer transformer, from scratch) ─► nutrients / 100 g

config.json records pretrained: false; model.load() rebuilds every network from config.json and loads model.safetensors, so inference never downloads a base model.

Usage

Python

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("shalev396/calorie-lens")
sys.path.insert(0, path)
import model
predictor = model.load(path, device="cpu")            # or "cuda"
result = predictor.predict("meal.jpg")                # a PIL image, bytes or base64 work too
result = predictor.predict("meal.jpg", total_grams=420)   # plate weighed? the scale becomes exact
result, heatmaps = predictor.analyze("meal.jpg")      # + where each ingredient's query looked
predictor.calculate([("grilled salmon", 150), ("white rice", 200)])   # stage 2b food calculator

Space API (free): the Space exposes /predict (photo + optional total grams in, [result, seconds, device] out).

Inference Endpoint: deploy this repo from its page (Deploy -> Inference Endpoints). handler.py takes an image (Content-Type: image/jpeg, or JSON {"inputs": "<base64>", "parameters": {"total_grams": 350}}) and returns the result above; {"inputs": [["oatmeal", 250], ["banana", 120]]} runs the calculator.

Training

• Data: shalev396/calorie-lens-data, built from Nutrition5k: one overhead RGB-D scan per dish where available and 5 frames from each of the 4 side-angle videos. Label bugs were fixed (Cafe 2 totals logged as 0, x1000 scale errors, empty names, duplicate table rows) and every dish was assigned a plate group (plates were scanned once per added item), so validation never shares a plate with training. Test = the official split.
• Recipe: see training/: AdamW with layer-wise learning-rate decay, warmup + cosine, bf16, EMA of the weights, early stopping on validation; augmentation on the GPU (any rotation for overhead scans, which keeps the camera-height scale cue).
• Selection: both backbones are trained with the same recipe; validation picks the winner (ties go to ConvNeXt), which is retrained with an auxiliary depth head. The test split is evaluated once, at the end.

Experiments

Validation picked ConvNeXt-B (17.22 vs 17.92). On the test split DINOv2-B came out lower (17.53 % vs 18.35 %), but the test split makes no choices, so ConvNeXt ships. The depth head (E3) did not help: it was slightly worse on validation and on test. The whole run (NutriText + 3 experiments) took 3.1 h on a Colab GPU.

Evaluation

Numbers: overhead test scans (the official depth_test split, labels corrected), as MAE and percent MAE (MAE / mean of the truth, the Nutrition5k convention). metrics.json also has the side-angle views, test_clean (plates never seen in training), the original labels and the 95 % CI.

Headline: 18.35 % calorie error (46.9 kcal) on the 507 overhead test scans, 95 % CI 16.3-20.2 % (bootstrap over plate groups). The original labels give the same 18.35 % on these scans, so the number compares directly with published results. Plate mass: 14.3 %. Averaging all side-angle frames of a dish: 17.3 %.

Small and medium plates are close: under 100 kcal the error is 11 kcal. Large plates are underestimated: over 600 kcal (38 dishes) the error is 167 kcal, because the model pulls toward a typical plate.

What each part is worth. The totals head read on its own scores 16.4 %, better than the 253 gram neurons (18.4 %): the itemised route, whose rows add up to the totals, costs about 2 points of calorie accuracy. Its estimate is returned as direct_estimate in every result. The plain sum of the soft grams, without rescaling to the plate mass, is the worst (23.6 %).

Ingredients: presence mAP 0.65 (F1 0.73 at the display threshold 0.65); 11.2 g error per present ingredient. Each ingredient's query looks at its own part of the plate:

NutriText (stage 2b), on 633 USDA foods it never saw: kcal R² 0.89 vs 0.78 for a TF-IDF nearest-neighbour lookup, 31.7 vs 47.4 kcal per 100 g, and better on all eight nutrients. On the 253 Nutrition5k names (zero-shot): 56 vs 87 kcal per 100 g (median 18 vs 49).

Limitations

• Every training photo comes from two Google cafeterias: a fixed overhead camera height and 253 ingredients. On other photos the mass scale is uncertain; entering the plate's weight (total_grams) removes that error completely.
• Hidden oil, butter, dressings and sauces can only be inferred from how the food looks.
• Only kcal, fat, carbs and protein are Nutrition5k's own values; the other four nutrients come from a USDA food matched to each ingredient. NutriText values for names it never saw are estimates, and some are off (it gives grilled salmon 7.7 g of carbs per 100 g).
• Plates over ~600 kcal are underestimated (see the chart above).
• Results on corrected labels are not directly comparable with published Nutrition5k numbers; the original-label results are in metrics.json.

Code licence: MIT. Training data: Nutrition5k (CC BY 4.0, Thames et al., CVPR 2021) and USDA FoodData Central (public domain).