FoodExtract-v1
This is a food and drink extraction language model built on Gemma 3 270M.
Given raw text, it's designed to:
- Classify the text into food or drink (e.g. "a photo of a dog" = not food or drink, "a photo of a pizza" = food or drink).
- Tag the text with one or more tags (see tags_dict below).
- Extract the edible food-related items as a list.
- Extract the edible drink-related items as a list.
For example, the input text might be:
British Breakfast with baked beans, fried eggs, black pudding, sausages, bacon, mushrooms, a cup of tea and toast and fried tomatoes
And the model will generate:
food_or_drink: 1
tags: fi, di
foods: British Breakfast, baked beans, fried eggs, black pudding, sausages, bacon, mushrooms, toast, fried tomatoes
drinks: tea
This model can be used for filtering a large image caption (e.g. DataComp-1B) text dataset for food and drink related items.
Dataset
The model was trained on the FoodExtract-1k dataset.
This dataset contains 1400 samples of raw text and JSON output pairs of structured food extractions provided by gpt-oss-120b.
For example, a raw image caption input might be:
another optional quest takes place on windfall island during the night time play the song of passing a number of times and each time, glance towards the sky
And the gpt-oss-120b generated output (JSON) would be:
{'is_food_or_drink': 'false', 'tags': [], 'food_items': [], 'drink_items': []}
This is condensed to:
food_or_drink: 0\ntags: \nfoods: \ndrinks:
Tags dictionary mapping
These tags are designed for fast filtering.
For example, the model can assign a certain tag based on what's in the raw text and then we can filter for "ingredient list" items.
tags_dict = {'np': 'nutrition_panel',
'il': 'ingredient list',
'me': 'menu',
're': 'recipe',
'fi': 'food_items',
'di': 'drink_items',
'fa': 'food_advertistment',
'fp': 'food_packaging'}
Helper functions
The model is trained to output a condensed version of the structured data.
We do this so the model can generate less tokens (e.g. it doesn't have to generate JSON outputs).
The following functions help to condense and uncondense raw text outputs/inputs into the desired structure.
def condense_output(original_output):
'''Helper function to condense a given FoodExtract string.
Example input: {'is_food_or_drink': True, 'tags': ['fi'], 'food_items': ['cape gooseberries', 'mulberry', 'chilli powder', 'flathead lobster', 'hoisin sauce', 'duck leg', 'chestnuts', 'raw quail', 'duck breast', 'rogan josh curry sauce', 'brown rice', 'dango'], 'drink_items': []}
Example output: food_or_drink: 1\ntags: fi\nfoods: cape gooseberries, mulberry, chilli powder, flathead lobster, hoisin sauce, duck leg, chestnuts, raw quail, duck breast, rogan josh curry sauce, brown rice, dango\ndrinks:'''
condensed_output_string_base = '''food_or_drink: <is_food_or_drink>
tags: <output_tags>
foods: <food_items>
drinks: <drink_items>'''
is_food_or_drink = str(1) if str(original_output["is_food_or_drink"]).lower() == "true" else str(0)
tags = ", ".join(original_output["tags"]) if len(original_output["tags"]) > 0 else ""
foods = ", ".join(original_output["food_items"]) if len(original_output["food_items"]) > 0 else ""
drinks = ", ".join(original_output["drink_items"]) if len(original_output["drink_items"]) > 0 else ""
condensed_output_string_formatted = condensed_output_string_base.replace("<is_food_or_drink>", is_food_or_drink).replace("<output_tags>", tags).replace("<food_items>", foods).replace("<drink_items>", drinks)
return condensed_output_string_formatted.strip()
def uncondense_output(condensed_output):
'''Helper to go from condensed output to uncondensed output.
Example input: food_or_drink: 1\ntags: fi\nfoods: cape gooseberries, mulberry, chilli powder, flathead lobster, hoisin sauce, duck leg, chestnuts, raw quail, duck breast, rogan josh curry sauce, brown rice, dango\ndrinks:
Example output: {'is_food_or_drink': True, 'tags': ['fi'], 'food_items': ['cape gooseberries', 'mulberry', 'chilli powder', 'flathead lobster', 'hoisin sauce', 'duck leg', 'chestnuts', 'raw quail', 'duck breast', 'rogan josh curry sauce', 'brown rice', 'dango'], 'drink_items': []}
'''
condensed_list = condensed_output.split("\n")
condensed_dict_base = {
"is_food_or_drink": "",
"tags": [],
"food_items": [],
"drink_items": []
}
# Set values to defaults
food_or_drink_item = None
tags_item = None
foods_item = None
drinks_item = None
# Extract items from condensed_list
for item in condensed_list:
if "food_or_drink:" in item.strip():
food_or_drink_item = item
if "tags:" in item:
tags_item = item
if "foods:" in item:
foods_item = item
if "drinks:" in item:
drinks_item = item
if food_or_drink_item:
is_food_or_drink_bool = True if food_or_drink_item.replace("food_or_drink: ", "").strip() == "1" else False
else:
is_food_or_drink_bool = None
if tags_item:
tags_list = [item.replace("tags: ", "").replace("tags:", "").strip() for item in tags_item.split(", ")]
tags_list = [item for item in tags_list if item] # Filter for empty items
else:
tags_list = []
if foods_item:
foods_list = [item.replace("foods:", "").replace("foods: ", "").strip() for item in foods_item.split(", ")]
foods_list = [item for item in foods_list if item] # Filter for empty items
else:
foods_list = []
if drinks_item:
drinks_list = [item.replace("drinks:", "").replace("drinks: ", "").strip() for item in drinks_item.split(", ")]
drinks_list = [item for item in drinks_list if item] # Filter for empty items
else:
drinks_list = []
condensed_dict_base["is_food_or_drink"] = is_food_or_drink_bool
condensed_dict_base["tags"] = tags_list
condensed_dict_base["food_items"] = foods_list
condensed_dict_base["drink_items"] = drinks_list
return condensed_dict_base
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
MODEL_PATH = "mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v1"
# Load the model into a pipeline
loaded_model = AutoModelForCausalLM.from_pretrained(
pretrained_model_name_or_path=MODEL_PATH,
dtype="auto",
device_map="auto",
attn_implementation="eager"
)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path=MODEL_PATH,
)
# Create model pipeline
loaded_model_pipeline = pipeline("text-generation",
model=loaded_model,
tokenizer=tokenizer)
# Create a sample to predict on
input_text = "A plate with bacon, eggs and toast on it"
input_text_user = [{'content': input_text, 'role': 'user'}]
# Apply the chat template
input_prompt = loaded_model_pipeline.tokenizer.apply_chat_template(conversation=input_text_user,
tokenize=False,
add_generation_prompt=True)
# Let's run the default model on our input
default_outputs = loaded_model_pipeline(text_inputs=input_prompt,
max_new_tokens=256)
# View the outputs
print(f"[INFO] Test sample input:\n{input_prompt}\n")
print(f"[INFO] Fine-tuned model output:\n{default_outputs[0]['generated_text'][len(input_prompt):]}\n")
You should see an output similar to:
[INFO] Test sample input:
<bos><start_of_turn>user
A plate with bacon, eggs and toast on it<end_of_turn>
<start_of_turn>model
[INFO] Fine-tuned model output:
food_or_drink: 1
tags: fi
foods: bacon, eggs, toast
drinks:
Training procedure
This model was trained with SFT (Supervised Fine-Tuning) via Hugging Face's TRL library.
See the full training walkthrough at: https://www.learnhuggingface.com/notebooks/hugging_face_llm_full_fine_tune_tutorial
Citations
- Reference for structured data extraction was taken from the paper Essential-Web v1.0: 24T tokens of organized web data and the subsequent fine-tuned model
EssentialAI/eai-distill-0.5b.