Qwen3.5-4B-APIGen-MT-5k

LoRA fine-tune of Qwen/Qwen3.5-4B on Salesforce/APIGen-MT-5k, merged to 16-bit weights. The goal was to improve multi-turn tool calling without materially degrading general reasoning benchmarks. All numbers below compare this checkpoint against the unmodified base model under the same evaluation setup.

Author: Colin Fitzsimons

Files

This repository contains both deployment formats from the same merged 16-bit checkpoint:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CFitzsimons/Qwen3.5-4B-APIGen-MT-5k-F16-GGUF"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

Results

Summary

The aggregate story is mixed rather than uniformly positive. BFCL mean accuracy moves less than half a percentage point. HellaSwag and ARC sit within each metric's reported stderr (±0.44–1.46%), so I would not read those shifts as meaningful on their own. The interesting signal is in the BFCL category breakdown, not the headline averages.

BFCL v4 (GGUF, llama-server)

Tool-calling eval on F16 GGUF via local llama-server. Categories: simple_python, irrelevance, parallel, multiple.

Parallel function calling improved by 9.5 pp. That is the clearest win and the one category where paired testing reached significance (p ≈ 0.003, McNemar on per-example pass/fail). Irrelevance detection dropped 4.2 pp (p ≈ 0.018), which suggests the model is somewhat more eager to invoke tools when it should abstain. Pooled BFCL accuracy did not reach significance (p ≈ 0.93).

BFCL parallel_multiple (GGUF, llama-server)

Combined parallel + multiple tool calls in one turn (200 cases). Clean full re-runs on both variants (single process, cache cleared).

Despite +9.5 pp on the isolated parallel category, the fine-tuned model regressed on the harder combined parallel_multiple suite. That suggests gains on parallel calls may not generalise when multiple distinct functions must be invoked correctly in a single response.

BFCL multi_turn_base (GGUF, llama-server)

Stateful multi-turn tool use (200 cases). Same GGUF / handler setup as above. Both scores are from clean full re-runs (single process, cache cleared).

The fine-tuned checkpoint regressed on multi-turn stateful tasks despite APIGen-MT-5k being multi-turn training data. That may reflect handler mismatch, empty-response failures on long trajectories, or overfitting to single-shot call patterns in the four-category suite.

HellaSwag (HF, full validation split)

ARC-Challenge (HF, full test split)

Evaluation assumptions

BFCL runs on GGUF because that matches the intended deployment path. HellaSwag and ARC run on Hugging Face weights: those tasks need token log-probabilities, and the current lm_eval GGUF backend does not parse the logprobs format returned by recent llama-server builds. The HF and GGUF artifacts come from the same merged checkpoint, but the inference stack differs between suites.

BFCL has no Qwen3.5-native handler in v4 yet. These runs used Qwen/Qwen3-4B-Instruct-2507-FC for prompt formatting and response parsing while serving Qwen3.5 weights locally. That is adequate for relative base-vs-fine-tuned comparison, but the absolute scores should not be treated as official Qwen3.5 leaderboard numbers.

Training

Fine-tuning used Unsloth for LoRA and TRL SFTTrainer on pre-rendered chat text. The base model loaded in 4-bit; adapters were merged to 16-bit after training.

Dataset: Salesforce/APIGen-MT-5k, split 90/10 train/eval with seed=3407.

Formatting: Each APIGen row is converted to Qwen3.5 chat format before SFT. System text comes from the system or policy field. Tool definitions are normalized (OpenAI-style function wrappers are unwrapped to plain {name, description, parameters} dicts). Turns map as follows: human → user, gpt → assistant, function_call → assistant with structured tool calls (JSON-parsed; think calls discarded), observation → tool response paired with the preceding call. The final training example is tokenizer.apply_chat_template(messages, tools=tools, tokenize=False), which injects tool schemas and call syntax consistent with Qwen3.5 inference.

Limitations

Training data is limited to APIGen-MT-5k. Generalization to unseen tool schemas is untested. The irrelevance regression on BFCL is the main behavioral concern: gains on parallel calls may come partly at the cost of knowing when not to call. BFCL handler mismatch adds noise to absolute scores. Benchmarks are English-only.

License

Inherits the license of Qwen/Qwen3.5-4B. APIGen-MT-5k is subject to its own dataset terms.

Citation

@misc{qwen35-4b-apigen-mt-5k,
  author       = {Colin Fitzsimons},
  title        = {Qwen3.5-4B fine-tuned on APIGen-MT-5k for edge tool use},
  year         = {2026},
  howpublished = {Hugging Face model repository}
}