amalia-desenrasca-9b

amalia-llm/AMALIA-9B-0626-SFT fine-tuned to call tools, with prompts in European Portuguese (pt-PT). Desenrascanço is the Portuguese art of improvising your way out of a mess.

🚀 Usage

The tools go in the system message. The model answers with <tool_call>{...}</tool_call>, or in plain text when no tool fits. Send the tool result back with the tool role.

import json

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jgalego/amalia-desenrasca-9b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")

tools = [
    {
        "name": "get_weather",
        "description": "Estado do tempo atual numa cidade",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "Nome da cidade"}},
            "required": ["city"],
        },
    }
]
system = "Tens acesso \u00e0s ferramentas abaixo. Para usar uma, responde apenas com \u003ctool_call\u003e{\"name\": \"nome\", \"arguments\": {\"argumento\": \"valor\"}}\u003c/tool_call\u003e. Se nenhuma ferramenta servir, diz com educa\u00e7\u00e3o que n\u00e3o podes ajudar."
system += "\n<tools>\n" + "\n".join(json.dumps(tool, ensure_ascii=False) for tool in tools) + "\n</tools>"
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "Está a chover no Porto?"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))
# Expected: <tool_call>{"name": "get_weather", "arguments": {"city": "Porto"}}</tool_call>

🏋️ Training

• Why: AMALIA's chat template has no tool support, and there is no public pt-PT tool-calling data. The tools go in the system message as plain text, and the template renders the tool role as is.
• Data: 2,700 conversations from jgalego/function-calling-pt-pt: 1,000 xLAM, 700 ToolACE, 400 Hermes and 600 When2Call refusals. Another 300 generated conversations use the demo's tools for parallel calls, relative dates, percentages, currencies and unsupported forecasts. User turns and final answers are pt-PT; tool schemas, calls and results stay in English.
• Pairs: Each assistant turn is one prompt/completion pair. Calls may be parallel. Only When2Call refusals are used because its follow-up questions often ask for information the user already supplied.
• Method: LoRA (r=16, alpha=32, all attention and MLP projections) on the bf16 model with Unsloth. 2 epochs, learning rate 2e-4 with a cosine schedule, effective batch size 16. The adapter is merged into the base model.

📊 Evaluation

BFCL pt-PT uses the BFCL v3 single-turn questions translated into pt-PT, with the original functions and answer keys. Scores are shares of questions answered correctly.

The second evaluation uses 200 held-out Glaive-derived conversations with pt-PT user turns: 96 structured call references and 104 without a structured call.

The last row is reference agreement, not a measure of whether no tool fits. A manual audit of its 104 rows found:

On the 41 missing-detail rows, 23 outputs explicitly request more information. Twelve call a tool instead; eleven use arbitrary defaults or placeholders and one infers Apple's AAPL ticker.

⚠️ Limitations

• Missing arguments: The model sometimes invents required values instead of asking the user. This happened on 11 of 41 audited missing-detail requests.
• Coordinates: It inferred city-center coordinates in two distance questions whose tools accepted coordinates only.
• Translated data: The pt-PT data is machine-translated by AMALIA itself, and nobody has reviewed it.
• English arguments: Values like city names follow the English source data, so expect "New York" rather than "Nova Iorque".