Llama-3.2-1B-Instruct — Function Calling (GRPO)
Code & writeup: https://github.com/keitake123/llama-function-calling-study
GRPO (RL post-training) applied to the v1 SFT model, aiming to recover the irrelevance-detection regression that SFT introduced. This is a negative result — reported honestly as part of a four-model study.
Results (BFCL v4)
| Category | v1 SFT | This model (GRPO) | Path B (refusal-SFT) |
|---|---|---|---|
| irrelevance | 5.8 | 5.4 | 21.7 |
| live_irrelevance | 16.9 | 17.2 | 36.8 |
| simple_python | 77.5 | 77.2 | 78.2 |
| multiple | 74.0 | 74.0 | 72.0 |
| live_simple | 57.0 | 56.2 | 54.3 |
| live_multiple | 38.8 | 39.4 | 35.7 |
| live_relevance | 93.8 | 93.8 | 81.2 |
Finding: GRPO with a verifiable, irrelevance-penalizing reward produced no measurable irrelevance recovery (5.8→5.4). Hypothesis: GRPO can only amplify behaviors the model already samples. The SFT "always-call" prior was so strong that the model essentially never sampled a refusal during rollouts, so all completions scored similarly and there was no gradient signal to reinforce refusal. Supervised demonstration (Path B) succeeded where RL did not.
An earlier GRPO attempt also revealed a reward-hacking failure mode: the model collapsed to degenerate repetitive output that exploited a gap in the reward function (unparseable output was scored as a valid refusal on irrelevance cases). This was fixed by hardening the reward with adversarial test cases before the final run.
Training details
- Base: v1 SFT model · Method: GRPO (TRL), LoRA
- Config: 500 steps, 16 generations/prompt, temperature 1.0, β (KL) 0.05, lr 1e-6
- Reward: partial credit for correct function name / arguments; explicit reward for a natural-language refusal on irrelevance cases (hardened against degenerate exploits)
- Prompts: 2000 (1200 xLAM call-required + 800 synthetic irrelevance)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("Keitsuna123/llama-3.2-1b-fc-grpo")
model = AutoModelForCausalLM.from_pretrained(
"Keitsuna123/llama-3.2-1b-fc-grpo", torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get the weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}
}}]
text = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
Part of a series
| Model | Description |
|---|---|
| fc-sft-full | v1 SFT on xLAM |
| fc-sft-v2-merged | SFT on xLAM + distilabel (data-scaling ablation) |
| fc-grpo | GRPO for irrelevance recovery (this model) |
| fc-pathb | Supervised refusal training (effective irrelevance fix) |
References & Acknowledgements
- Base model: Llama 3.2 (Meta AI) — meta-llama/Llama-3.2-1B-Instruct
- Training data: Salesforce xLAM — xlam-function-calling-60k
- Benchmark: Berkeley Function Calling Leaderboard (BFCL) — Gorilla project
- Frameworks: HuggingFace TRL (GRPOTrainer), PEFT, Transformers
Citation
@misc{taketsuna2026_fc_smallmodel,
title = {Small-Model Function Calling: Comparing SFT, Data Scaling, GRPO, and Supervised Refusal Training},
author = {Taketsuna, Keiichi},
year = {2026},
howpublished = {\url{https://huggingface.co/Keitsuna123}},
note = {Llama-3.2-1B function-calling post-training study on BFCL v4}
}