ultravox-v0_6-qwen-3-32b-arabic
fixie-ai/ultravox-v0_6-qwen-3-32b with its Whisper
audio tower adapted for dialectal Arabic using
oddadmix/whisper-large-v3-turbo-arabic-dialectal-v2.
This repository is self-contained: the Qwen3-32B language model, the merged audio tower, the multimodal projector, the tokenizer and the Whisper feature extractor all ship here, so no other repo is fetched at load time.
How the audio tower was built
| Component | Source |
|---|---|
language_model.* | Qwen/Qwen3-32B (bf16, 64 layers) |
audio_tower.* | delta merge described above (Whisper-large-v3-turbo encoder, 32 layers) |
multi_modal_projector.* | fixie-ai/ultravox-v0_6-qwen-3-32b (finetuned) |
| tokenizer / chat template | fixie-ai/ultravox-v0_6-qwen-3-32b (Qwen3) |
audio_processor/ | openai/whisper-large-v3-turbo feature extractor |
Usage
import librosa, torch, transformers
repo = "ISTNetworks/ultravox-v0_6-qwen-3-32b-arabic"
processor = transformers.AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = transformers.AutoModel.from_pretrained(
repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto"
)
audio, sr = librosa.load("sample.wav", sr=16000)
turns = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "استمع للصوت وأجب باللغة العربية.\n<|audio|>"},
]
text = processor.tokenizer.apply_chat_template(
turns, add_generation_prompt=True, tokenize=False, enable_thinking=False
)
inputs = processor(text=text, audio=audio, sampling_rate=sr, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Audio is inserted with the <|audio|> placeholder in the user turn. Inference needs ~67 GB of bf16 weights
(2×80 GB GPUs, or one 80 GB GPU with quantization / CPU offload).
Notes and caveats
- The fine-tune: training was performed on the LLM