🩺 Docto Decision Qwen3.5-2B (v0.1)

✨ Summary

Docto Decision is a family of French medical decision models. You give a text (a patient message, a clinical note, a case report), a question and the possible answers, and the model returns one probability per answer.

This model is a LoRA fine-tune of Qwen3.5-2B (2.27B parameters), merged into the base weights (the adapter is in adapters/). It answers with a single token, an option letter or « oui » / « non », and the probabilities are read from that token.

The task is defined at inference time by the instruction, the question and the answers. Tasks close to the training ones work best, and new tasks can be tried zero-shot.

🎯 Use cases

🚀 Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bofenghuang/docto-decision-qwen3.5-2b-fr-v0.1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval().to("cuda")
SYSTEM = ("Tu es un expert médical francophone. On te donne une tâche de décision : une consigne, éventuellement un profil, "
          "un contexte et un élément, une question et les réponses possibles. Résous la tâche. Réponds uniquement par la réponse "
          "demandée, sans explication.")


def decide(task, question, options, context=None, profile=None, item=None, kind="choice"):
    """options: [(key, definition)]. kind: "choice", "score" (levels in order) or "noul" (one option: the proposition)."""
    parts = [f"[TASK] {task}"]
    if profile:
        parts.append(f"[PROFILE] {profile}")
    if context:
        parts.append(f"[CONTEXT]\n{context}")
    if item:
        parts.append(f"[ITEM] {item}")
    parts += [f"[QUESTION] {question}", "[OPTIONS]"]
    if kind == "noul":
        parts += [f"oui : {options[0][1]}", "non : ce n'est pas le cas.", "", "Réponds uniquement par « oui » ou par « non »."]
        answers = ["oui", "non"]
    else:
        answers = list("ABCDEFGHIJKLMNOPQRSTUVWXYZ"[: len(options)])
        parts += [f"{a}) {d}" for a, (_, d) in zip(answers, options)]
        parts += ["", f"Réponds uniquement par la lettre de la bonne réponse ({', '.join(answers)})."]
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "\n".join(parts)}]
    text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
    with torch.no_grad():
        logits = model(ids).logits[0, -1].float()
    first = [tok.encode(a, add_special_tokens=False)[0] for a in answers]  # first token of each answer
    p = torch.softmax(logits[first], -1)
    if kind == "noul":
        return p[0].item()  # P(oui) = P(true)
    return dict(zip([k for k, _ in options], p.tolist()))


decide(task="Déterminer la conduite à tenir la plus appropriée face à ce message.",
       profile="Patient : homme, 58 ans. Antécédents : hypertension.",
       context="Patient : Bonjour, j'ai une douleur dans la poitrine qui serre depuis 30 minutes et qui descend dans le bras gauche.",
       question="Quel niveau d'urgence correspond à cette situation ?",
       options=[("home_care", "Pas d'urgence : auto-surveillance à domicile chez un adulte sans fragilité, ou demande sans symptôme."),
                ("consult_48h", "Consultation dans les 48 heures : médecin traitant, sans caractère de gravité mais qui dure ou gêne."),
                ("same_day", "Consultation le jour même : signe préoccupant sans détresse vitale."),
                ("emergency_now", "Urgence immédiate : appeler le 15 ou se rendre immédiatement aux urgences.")],
       kind="score")
# -> {'home_care': 0.000, 'consult_48h': 0.000, 'same_day': 0.002, 'emergency_now': 0.998}

decide(task="Dire si le signe indiqué est décrit comme présent chez l'enfant dans la conversation.",
       profile="Enfant : bébé de 4 mois, garçon",
       context="Parent : Bonjour, mon fils a 39 de fièvre depuis ce matin, il ne boit presque plus ses biberons et il est très mou.",
       item="Signe : nourrisson de moins de 6 mois qui boit nettement moins que d'habitude",
       question="Ce signe est-il présent d'après le parent ?",
       options=[("present", "Oui, le parent décrit ce signe comme présent.")],
       kind="noul")
# -> 0.65

• Write each option as a full French sentence: the model reads the definitions. For choice, option order does not matter. For score, keep the levels in order.
• For noul, pass a single option (the proposition) and read P(true).
• The text is French and the tags ([TASK], [CONTEXT]...) are English. Leave out empty fields. Up to 26 options (letters A to Z).
• For throughput, serve it with vLLM, as the benchmark does.

📊 Evaluation

Results on the Docto Decision Bench: 12 decision tasks on real French patient messages and two clinical datasets (DiaMed, MORFITT). The benchmark repository has the data and the evaluation code.

This model on each task:

• All scores are percentages. Labels: silver = Qwen3.8-27B and Qwen3.5-122B-A10B agree on the answer, gold = human labels of the dataset, audit = blind relabelling by another model.
• OpenJev is a fine-tune of Qwen3.8-27B, one of the two teachers behind the silver labels, so on the 9 silver tasks it is partly scored against its own teacher's answers.
• 95% CIs from 1,000 bootstrap resamples of the items in each task. Order consistency: share of answers that stay the same when the options of the 6 choice tasks are shown in 3 different orders.
• Latency: median time for one request, on one H100 in BF16 (OpenJev in FP8), without inference optimisation. Jev and Gemini are timed from the client, network included.

🏋️ Training

LoRA (rank 16, alpha 32, attention and MLP projections of the language model) on Qwen3.5-2B, 1 epoch at learning rate 1e-4 on docto-decision-data-fr-v0.1 (943,013 decision records, 108 tasks). The target of each record is half the correct answer and half the answer distribution of Qwen3.8-27B.

⚠️ Limitations

• This is a research model, not a medical device, and it has not been validated clinically. Keep a human in the loop.
• It can under-estimate urgency. Do not use it as the only safety net for triage.
• It is weaker on tasks far from the training tasks, and on medical knowledge questions it stays well below large LLMs.
• Most training labels come from one model family (Qwen), so the model shares its conventions and its errors.
• French only. Check calibration on your own task before choosing a threshold.