Qwen3-Omni-30B-A3B-Thinker-bf16

The Thinker of Qwen/Qwen3-Omni-30B-A3B-Instruct, QLoRA fine-tuned on SLURP intent classification and merged back to bfloat16. It scores 86.80% (434/500) on the SLURP test protocol described below, against 79.40% (397/500) for the un-fine-tuned base checkpoint.

This repository is the bfloat16 reference for a quantization campaign. It is the arm every quantized arm is measured against, and it is the checkpoint you start from if you want to quantize the model yourself.

Which of the four repositories do you want? If you just want to run the model, take the 4-bit sibling — 14.77 GiB, 86.20%, does not separate from this repository, fits one 24 GB card, and needs pip install dynquant plus one registration call. Take this repository if you want the reference arm, or an input to quantize yourself. The adapter is for rebuilding this checkpoint; the 3-bit repository is a record of a measured collapse and is not a model.

Read this before you download 59 GiB: text output only, no Talker

Qwen3-Omni is an any-to-any model that can answer in speech. This checkpoint cannot.

Only the Thinker was trained, merged and published. The Talker and code2wav stacks (3,540,613,057 parameters) were nulled out before training and are absent from this repository. The consequence is simple and total:

• Input: audio, images, video and text.
• Output: text, and nothing else. There is no speech output. There is no audio decoder in this repository and no way to enable one.

If you came here expecting to speak to the model and hear it answer, this is the wrong repository; use Qwen/Qwen3-Omni-30B-A3B-Instruct instead.

Concretely:

• Class: Qwen3OmniMoeThinkerForConditionalGeneration
• model_type: qwen3_omni_moe_thinker
• config.json holds exactly three sub-configs: audio_config, text_config, vision_config. There is no talker_config.
• AutoModelForCausalLM and AutoModel both fail. Neither mapping carries a qwen3_omni_moe_thinker entry, so both raise ValueError: Unrecognized configuration class Qwen3OmniMoeThinkerConfig. Load Qwen3OmniMoeThinkerForConditionalGeneration explicitly, or AutoModelForImageTextToText, which is the auto class this model_type is registered under. All of these ship natively in transformers 5.15.0 — no trust_remote_code anywhere.

Model facts

The directory ships processor_config.json and no preprocessor_config.json. That is correct and AutoProcessor resolves from it; do not add the missing-looking file. If you ever see an OSError naming preprocessor_config.json, that filename is a red herring — the file that actually decides resolution is processor_config.json.

Size

Three different numbers get quoted as "the size of a checkpoint". All three are given so nothing here has to be inferred, and the GiB conversion is attached to the level it belongs to:

Requirements

The weights are 59.08 GiB, so plan on a single 80 GiB card (A100 80GB, H100, H200): that leaves roughly 20 GiB for activations and KV cache, which is ample for clips of this length. A 48 GiB or 64 GiB card cannot hold it — shard with device_map="auto" across two or more devices instead. Or take the 4-bit sibling, whose measured resident VRAM is 15,892,454,912 B (14.80 GiB) and which therefore fits one 24 GB card — but pip install dynquant and call dynquant.register_hf_quantizer() before you load it. Without both, transformers does not raise: it logs a warning, sets pre_quantized=False, and hands back a randomly initialised model that scores at chance.

This repository itself needs nothing beyond transformers. It is plain bfloat16.

Loading and running it

Needs transformers >= 5.0. The snippet below passes dtype=, which is the v5 spelling of torch_dtype=; on transformers 4.x that keyword is silently swallowed, the model loads in float32 at ~127 GB, and the only symptom is an OOM. Measured on transformers 5.15.0 / torch 2.11+cu128.

pip install "transformers>=5.15,<6" accelerate torch
# plus an audio decoder backend if you feed the processor a path or a URL rather than an array:
pip install torchcodec        # transformers prefers this when present
# or:  pip install librosa soundfile

import torch
from transformers import AutoProcessor, Qwen3OmniMoeThinkerForConditionalGeneration

model_id = "VikramPal/Qwen3-Omni-30B-A3B-Thinker-bf16"

processor = AutoProcessor.from_pretrained(model_id)
# AutoProcessor has no mapping for model_type `qwen3_omni_moe_thinker`; it resolves through the
# `processor_class` field in processor_config.json. If that field goes missing -- a partial
# clone, a hand-edited file -- it does not raise, it falls back to a bare tokenizer, which drops
# the audio without error. Assert what you got:
assert type(processor).__name__ == "Qwen3OmniMoeProcessor", type(processor).__name__

model = Qwen3OmniMoeThinkerForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,   # `torch_dtype` on transformers < 5
    device_map="auto",      # needs `accelerate`
)
model.eval()

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Listen to the spoken command and classify it into one of\n"
                                     "these intents.\nAnswer with the number only.\n"
                                     "0. alarm_query\n1. alarm_remove\n...\n59. weather_query"},
            # This checkpoint was fine-tuned to answer with an INDEX into a 60-line numbered
            # menu. Prompt it without the menu and it still answers with a bare integer, which
            # then names nothing. Build the menu as described under "Results" below.
            {"type": "audio", "audio": "utterance.wav"},
            # A path or URL is decoded and resampled to 16 kHz for you. A numpy array is NOT:
            # `load_audio` returns it untouched, so it must already be mono float32 at 16 kHz.
            # A stereo or 44.1 kHz array raises nothing -- the encoder reads the clip at the
            # wrong speed and the model scores between chance and correct.
            {"type": "text", "text": "Intent:"},
        ],
    },
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    # `do_sample=False` alone is not enough on transformers 5.x: unset generation fields are
    # filled from the checkpoint's own generation_config, so a shipped repetition_penalty
    # survives and moves the score silently. Pin the neutral fields.
    out = model.generate(
        **inputs,
        max_new_tokens=8,
        do_sample=False,
        num_beams=1,
        repetition_penalty=1.0,
        length_penalty=1.0,
        no_repeat_ngram_size=0,
    )

answer = processor.batch_decode(
    out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer)   # e.g. "37"  ->  intents[37]

generate returns text token ids only — there is no audio return value to unpack, and asking the call for audio is meaningless here.

Results

Every arm below is the first 500 items of a random.Random(0) shuffle of the SLURP test split (13,078 recordings joined to SLURP's annotation first). The shuffle is load-bearing, not cosmetic: the mirror arrives grouped by speaker and scenario, so the raw first 500 rows sample a handful of intents and read as a destroyed model. 4 shots drawn from train, greedy decoding, max_new_tokens=8, stop at the first newline, add_special_tokens=False, max_prompt_tokens=4096, batch size 8, MoE experts dispatch pinned to eager (the checkpoint's own choice is grouped_mm; pinning keeps every arm computing the same thing), and a fixed 60-intent menu (intents_sha d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d). Chance is 1.667%. Per-item hits are stored, so every comparison is an exact McNemar test on paired outcomes rather than a difference of two independent proportions.

95% confidence intervals on the paired differences: base to sft [+4.51, +10.29], sft to dq4 [−2.64, +1.44], sft to dq3 [−66.17, −57.43]. The omni-dq3 arm returned 3 unparseable generations; every other arm in this table returned 0.

The prompt, exactly

The figures above were produced with a 4-shot prompt that lists the 60-intent menu and asks for an index into it, not an intent name. The scorer accepts only an integer in [0, 60); anything else is unparseable and counts as wrong. The scored prompt is exactly this, as one user turn in this order:

Listen to the spoken command and classify it into one of these intents.
Answer with the number only.
0. alarm_query
1. alarm_remove
... 60 lines, one per intent ...
59. weather_query

Examples, as text:
<gold transcript of shot 1> -> <its index>
<gold transcript of shot 2> -> <its index>
<gold transcript of shot 3> -> <its index>
<gold transcript of shot 4> -> <its index>

Now the spoken command:
[audio]
Intent:

• The menu is sorted(set(f"{scenario}_{action}")) over the union of train, devel and test of SLURP's own annotation — all three splits, because devel and test are each missing a different intent and a per-split menu would renumber the taxonomy. sha256 of the menu joined by newlines is the intents_sha above.
• The shots are drawn from train, as text transcripts, never as audio: sorted(random.Random(0).sample(range(50628), 4)) over the random.Random(0)-shuffled train pool, rendered one per line as {transcript} -> {index}. Their gold indices are 27, 15, 34, 22, and those four rows are excluded from the 8,000 training rows.
• The clip goes into the content block as a bare mono float32 np.ndarray at 16 kHz — the {"array", "sampling_rate"} dict raises — with the measured rate passed beside it as a processor kwarg. A wrong rate raises nothing; it reads the clip at the wrong speed and scores between chance and correct.
• Scoring strips any reasoning trace, then takes a leading integer in [0, 60) if there is one, else the first in-range integer anywhere in the generation.

Do not hand-build any of this. The harness that produced every number on this page ships in the same package, and it registers the DynQuant quantizer itself and raises if a packed checkpoint comes back unpacked:

pip install dynquant "transformers>=5.15,<6" "datasets>=4" torchcodec

dynquant eval VikramPal/Qwen3-Omni-30B-A3B-Thinker-bf16 \
  --task slurp --split test --limit 500 --shots 4 --shot-seed 0 \
  --model-class Qwen3OmniMoeThinkerForConditionalGeneration \
  --experts-impl eager --device cuda:0 --dtype bfloat16 --out sft.json

Reproducing it downloads the whole SLURP test split and the whole train split (50,628 recordings, pulled in full to draw four text exemplars) and fetches labels from raw.githubusercontent.com, so budget the disk and the network.

Three readings, stated no more strongly than the numbers allow

The fine-tune worked. +7.40 points over the base checkpoint, separated at p = 7.51e-07. Note that this one comparison differs in two things at once: the base arm ran the whole Omni checkpoint while every post-merge arm runs the Thinker alone, and only the post-merge arms carry the merged adapter. It remains a fair comparison, since the Talker is strictly downstream of the text the Thinker emits and SLURP scores that text, but it is not a one-variable contrast the way the quantization arms are.

4 bits costs nothing this experiment can detect. The correct claim is the interval: at 4.00x fewer bytes the comparison excludes damage worse than 2.64 points. It does not establish that damage is zero.

3 bits destroys the model, for a reason that is known. DynQuant's role floors alone cost 3.418 average bits on this architecture, so a 3.00-bit target sits below the floor budget. The 3-bit arm measures floor-override damage; it is not "the 4-bit experiment at a lower budget". The detail is on that repository's card.

Task and data

SLURP intent classification. The label is the joint scenario_action pair, a 60-class menu.

Audio comes from the Hub mirror marcel-gohsen/slurp. Labels do not: they are taken from SLURP's own annotation at https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl. The reason is worth one sentence, because it will bite anyone who trains on the mirror's own columns: the mirror's intent field is corrupted on 1,548 of 72,396 recordings, where it disagrees with the scenario/action pair by having dropped the scenario, so nine distinct *_query intents all collapse to query. Taken at face value that field yields 91/71/77 classes across the three splits against the real 60. The scenario and action columns are clean, so the joint pair is the label used here.

Training

QLoRA over an NF4-quantized base, peft 0.20.0.

The MoE expert banks were not adapted. LoRA does not reach batched 3-D expert banks. Under LoRA every base weight is frozen; only rank-16 adapters on 384 projections took a gradient, and the 91.399% of parameters in the 96 expert banks carried no adapter at all. The merge touched 384 weights; the 96 expert banks are unchanged. What was fine-tuned here is attention, the dense projections around it and the audio tower, and the +7.40 points come from that alone.

The adapter was trained against an NF4 base and merged into bfloat16. That is standard for QLoRA, and it means this merged checkpoint is not bit-for-bit the model that produced the training-time statistics.

To rebuild this checkpoint from the adapter, see the adapter card: load Qwen/Qwen3-Omni-30B-A3B-Instruct with the full Qwen3OmniMoeForConditionalGeneration class and take .thinker out of it, then PeftModel.from_pretrained(thinker, adapter).merge_and_unload(safe_merge=True), and run the save on CPU — transformers v5's save_pretrained OOMs beside its own model on GPU.

Related repositories

Four artefacts from one campaign; all four cross-link to each other.

Both quantized siblings are Thinker-only and text-out as well: neither contains a Talker, so neither can produce speech.

If you load either quantized sibling, pip install dynquant is required and is not sufficient on its own — you must also call dynquant.register_hf_quantizer() before from_pretrained, because registration is an explicit call and deliberately not an import side effect. Without it transformers does not raise: it logs a warning, silently sets pre_quantized=False, and returns a randomly initialised model that generates fluent garbage and scores at chance. Those cards say so at the top. None of it applies to this repository, which is plain bfloat16.

Quantizing it yourself

This checkpoint is the input to both quantized arms. Those were produced with DynQuant 0.4.0 at group size 128, asymmetric (a per-group bfloat16 scale and offset), layout linear, compute dtype bfloat16, covering 650 modules and 31,712,079,872 elements.

7,125,616 dense parameters (norms, biases, convolutions) stay in bfloat16. The 48 MoE routers (mlp.gate) are a third case, and the one most often misstated: they are in the map at 8 bits and are written packed at 8 bits (270,336 B each, inside the packed total rather than beside it). Nothing in the packed runtime can stand where a router stands, so the loader dequantizes them back into a dense bfloat16 weight after load — which is why a read-back reports 48 of 48 routers dense. Their values therefore carry 8-bit quantization error; only their storage at runtime is dense. What the floor policy guarantees is that a router is never breached: MOE_ROUTER is a structural role, so 8 bits is what it keeps at both budgets, and the 3-bit collapse is not a routing collapse.

Nothing about this repository is specific to DynQuant. It is an ordinary bfloat16 transformers checkpoint, and any quantizer that handles a Qwen3-Omni Thinker will read it.

Limitations

91.399% of the parameters were never adapted. The 96 batched MoE expert banks hold 28,991,029,248 of 31,719,205,488 parameters and LoRA does not reach them. Whatever this model knows about SLURP, it learned in attention, the dense projections and the audio tower. Anyone expecting a fine-tune to have reshaped the expert mixture should expect the opposite.

500 evaluation items bound the resolution. That is the entire basis for every accuracy figure here. It is enough to separate +7.40 points at p = 7.51e-07, and enough to place the 4-bit comparison inside [−2.64, +1.44]. It is not enough to resolve anything smaller than a point or two, and no result on this card should be read to more precision than its interval.

One task, one language. SLURP intent classification, English only, under a fixed 4-shot prompt with the full 60-intent menu in context. Nothing here measures the model's general capability, its behaviour in any other language, or the multimodal skills it inherited from the base checkpoint — which the fine-tune may well have degraded, since nothing tested them before or after. If you need the base model's general behaviour, use the base model.

SLURP is one collection with one recording protocol. Accuracy on it does not transfer unexamined to other microphones and acoustic conditions, telephony audio, unfamiliar accents, or spontaneous speech outside this collection.

59 of 60 intents were seen in training. One class of the label menu never appeared in the 8,000 sampled training recordings.

No competitive baselines. GPTQ, AWQ and RTN were out of scope for this phase, so there is no matched-bytes comparison against another quantizer anywhere in these four repositories.

Text output only. Stated once more, because it is the thing most likely to be discovered too late: no Talker, no code2wav, no speech.

Licence

The weights inherit the base model's licence. Mirroring Qwen/Qwen3-Omni-30B-A3B-Instruct: license: other, license_name: apache-2.0. Consult the base model repository for the terms that govern use of the underlying weights.

SLURP is distributed under CC BY 4.0. The fine-tuning data and the labels derive from it.

Citation

SLURP:

@inproceedings{bastianelli-etal-2020-slurp,
  title     = {{SLURP}: A Spoken Language Understanding Resource Package},
  author    = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2020}
}

Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct (Qwen team).

For the quantization work these checkpoints come from, cite the DynQuant repository: https://github.com/kambojvikram/dynquant, version 0.4.0.