Qwen3-4B-OT3-tau

Qwen3-4B-OT3-tau is MMOPD/Qwen3-4B-OT3-2ep continued with supervised fine-tuning on the Tau2 tool-use trajectories from inclusionAI/AReaL-tau2-data.

The starting point is a thinking model — every answer opens with a <think> block — that was itself Qwen/Qwen3-4B-Base fine-tuned on OpenThoughts3-1.2M. This checkpoint adds multi-turn, tool-calling agent behaviour on top of that reasoning ability while keeping the native chat template unchanged.

This checkpoint is released as one of a matched pair: MMOPD/Qwen3-4B-OT3-tau and MMOPD/Qwen3-1.7B-OT3-tau were trained on the same data with identical hyperparameters, so they differ only in model size.

Training data

33,531 Tau2 trajectories (tau2_sft_train.jsonl), every trajectory included regardless of task outcome. Each example is one assistant turn — reasoning, visible content, and tool calls — conditioned on the full preceding conversation.

Preprocessing:

• answer.thinking is renamed to reasoning_content, the field the native template reads.
• Tool-call-only messages that omit content are normalised to content: "", since the template accesses message.content directly.
• Histories longer than the 32,768-token context keep the system policy and the complete interaction from the final real user query onward. If still too long, only the largest tool-result payloads are shortened, retaining their head and tail. The final user request and the target turn are never shortened.

Loss masking

The native chat_template.jinja has no {% generation %} blocks, so assistant-only masks cannot be derived from it. Labels are instead built by rendering each example twice — the context with add_generation_prompt=True, and the full conversation with the target appended — and requiring the first token sequence to be an exact prefix of the second. Prefix tokens get label -100; everything after the boundary is supervised:

<think>
REASONING
</think>

CONTENT OR TOOL CALLS<|im_end|>

So loss covers the opening <think>, the reasoning, the closing </think>, the visible content or tool calls, and the final <|im_end|> — training the model to stop. The assistant header, system policy, user turns, earlier assistant turns and tool responses are all masked.

Training procedure

Trained with TRL SFTTrainer on pretokenized input_ids/labels. Released weights are bf16, matching the base snapshot.

Usage

The chat template is unchanged from MMOPD/Qwen3-4B-OT3-2ep, including its thinking behaviour.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MMOPD/Qwen3-4B-OT3-tau"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Book me a flight to Seoul next Tuesday."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=1024)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False))

Pass tools through apply_chat_template(..., tools=[...]) to use the tool-calling behaviour this checkpoint was trained for.

Note that config.json carries use_cache: false, inherited from the base snapshot. Set use_cache=True when generating with transformers for the usual KV-cache speedup; vLLM manages its own cache and is unaffected.

Limitations

Trained on all Tau2 trajectories regardless of task success, so it imitates unsuccessful as well as successful tool-use episodes. It inherits the base model's limitations and has not been safety-tuned beyond whatever the base and the Tau2 data provide. No held-out Tau2 evaluation is reported here.

License

Apache-2.0, following MMOPD/Qwen3-4B-OT3-2ep and the Qwen3 base models.