Sunflower Qwen3.5 9B Tool-Calling SFT PoC

Experimental full-parameter continuation of Sunbird/Sunflower-Qwen3.5-9B for multilingual external-LLM routing.

The model receives a request in a supported language and chooses whether to answer directly or emit a query_external_llm call containing an English version of the request. When it calls the tool, the host executes the request, returns the English result, and invokes the model again for a localized final answer.

This is an experimental checkpoint for SunbirdAI/sunflower#42, following SunbirdAI/sunflower-data-preparation#18. The interface works end to end, but the current checkpoint is not release-ready: it calls the tool too often and requires another routing-policy iteration.

Tool contract

The inference host must expose this JSON Schema when applying the chat template:

{
  "type": "function",
  "function": {
    "name": "query_external_llm",
    "description": "Send an English prompt to a strong external language model.",
    "parameters": {
      "type": "object",
      "properties": {
        "prompt": {
          "type": "string",
          "description": "The complete request in English."
        }
      },
      "required": ["prompt"]
    }
  }
}

The weights generate the call but do not execute it. The host must:

1. Parse the generated call.
2. Send prompt to a local or cloud model.
3. Append the English response as a role: tool message.
4. Invoke Sunflower again to generate the localized answer.

How tool calling works

Localized user request
        |
        v
Sunflower emits query_external_llm(prompt: string)
        |
        v
Host sends the English prompt to the external model
        |
        v
Host appends query_external_llm(response: string)
        |
        v
Sunflower answers in the user's language

Illustrative Luganda exchange

This example uses the exact interface and message shapes used in training. The wording is illustrative rather than a claimed held-out generation.

User (role: user)

Mu tterekero mulimu obusanduuko 17, era buli kasanduuko kalimu eccupa 24.
Eccupa 289 bwe zitundibwa, zisigala mmeka?

Assistant tool call — canonical structured message

{
  "role": "assistant",
  "content": "",
  "tool_calls": [
    {
      "type": "function",
      "function": {
        "name": "query_external_llm",
        "arguments": {
          "prompt": "In a warehouse there are 17 boxes, and each box has 24 bottles. After 289 bottles are sold, how many bottles remain?"
        }
      }
    }
  ]
}

Qwen renders and generates the same call in its model-specific textual form:

<tool_call>
<function=query_external_llm>
<parameter=prompt>
In a warehouse there are 17 boxes, and each box has 24 bottles. After 289 bottles are sold, how many bottles remain?
</parameter>
</function>
</tool_call>

Tool result appended by the host (role: tool)

{
  "role": "tool",
  "name": "query_external_llm",
  "content": "{\"response\":\"17 × 24 = 408 bottles. 408 − 289 = 119 bottles remain.\"}"
}

Final assistant response (role: assistant)

Obusanduuko 17 × eccupa 24 = eccupa 408. Bwe tuggyako eccupa 289,
eccupa 119 ze zisigala.

For a direct request, the model skips the tool exchange and immediately emits a localized assistant response, even though the tool remains available.

Training

• Dataset: Sunbird/sunflower-posttrain-data, config sft_tool_calling
• Dataset revision: a2cfa0b9b834efe0c013778c6802ddf4ad2a5955
• 27 languages: 26 Uganda-focused languages plus English
• Explicit source-isolated train and test splits
• 864 training rows and 216 test rows
• Target routes balanced 50/50 between tool and direct in each language
• 25% multi-turn rows
• One epoch of full-parameter BF16 SFT over all 9.41B parameters
• 8-bit AdamW, learning rate 1e-5
• Micro-batch 1, effective batch 16, 54 optimizer steps
• Maximum sequence length 2,304; longest row 1,692; zero rows dropped
• Hardware: one NVIDIA RTX PRO 6000 Blackwell Max-Q GPU
• Final train loss: 0.8320; eval loss: 0.8878

Five languages were excluded because too few translations survived quality control: bfa, kdj, kpz, luc, and mhi.

V2 evaluation

Deterministic greedy evaluation used the source-isolated 216-row test split: 108 tool-positive and 108 direct examples. For valid tool-positive calls, the host replayed the stored English result before asking the model for its final localized answer. No live external API was called.

Diagnostic routing slices

The model learned the tool interface more successfully than the routing policy: almost every emitted call is structurally valid and every valid call uses the correct function, but 67 of 108 direct-labelled examples still emitted a tool call.

Some measured overcalling comes from supervision problems. Broad seed rules labelled requests beginning with words such as write, create, or hello as direct even when the rest of the request asked for code or factual analysis. The test distribution is also skewed toward creative requests and contains no tool_to_direct or tool_to_tool histories. These labels and splits need to be corrected before the next training iteration.

Final-answer chrF is reported for reproducibility but should not be treated as a reliable quality measure for open-ended creative or conversational answers.

Historical v1 result

An earlier dataset revision used a positional 104-row holdout and achieved routing F1 0.9346, direct specificity 0.8868, and valid-call rate 1.0. That result proved the protocol could be learned, but the split was not source-isolated and is not directly comparable with the harder v2 evaluation.

Prompt construction

Use the tokenizer's native chat template, supply the tool schema, and disable thinking:

from transformers import AutoTokenizer

model_id = "ak3ra/sunflower-qwen3.5-9b-tool-sft-poc"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

query_external_llm_schema = {
    "type": "function",
    "function": {
        "name": "query_external_llm",
        "description": "Send an English prompt to a strong external language model.",
        "parameters": {
            "type": "object",
            "properties": {"prompt": {"type": "string"}},
            "required": ["prompt"],
        },
    },
}

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "<request in the user's language>"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=[query_external_llm_schema],
    enable_thinking=False,
    tokenize=False,
    add_generation_prompt=True,
)

See sunflower/tool_eval.py and scripts/eval_tool_calling.py for parsing and two-stage replay examples. The full frozen experiment report is in reports/tool-calling-poc/REPORT.md.

Next iteration

The proposed next step is cost-aware reinforcement learning:

reward = answer quality - lambda * tool calls

Easy requests can earn the highest reward by answering directly. Difficult requests can pay the small tool-call cost when delegation improves the answer. Before RL, the routing labels, per-task balance, and multi-turn test coverage need to be corrected.

Limitations

• This is a proof-of-concept checkpoint, not a production release.
• Routing is over-eager and fails the agreed acceptance gates.
• The current dataset covers 27 rather than all 32 intended languages.
• Some direct/tool labels need policy correction.
• The staging repository name is mutable. The evaluated weight revision before this documentation update was 837715f226d8f5bfc94bebaba702558495da4d61.
• A deployment host must implement tool execution, safety policy, timeouts, retries, authentication, observability, and cost controls.