Leveraging the multi-dimensional fine-grained annotations produced by our pipeline, we introduce FM-Speech, built upon the frontier Qwen3-Omni (30B MoE) architecture.
🎙️ Input: Raw Speech Audio ➔ 📊 Output: 14-Dimension Fine-Grained Speech Attributes (Structured JSON)
To overcome modality gaps and text-conditioned hallucinations, FM-Speech is trained using a Progressive Curriculum Fine-Tuning framework, decoupling complex auditory comprehension into three incremental stages: Warm-up (MCQ/QA) --> Capability Ramp-up --> Final Alignment (Full JSON).
🚀 Usage & Environment Setup
Our model is built upon the Qwen3-Omni architecture. We strongly recommend using vLLM for the inference and deployment of FM-Speech.
Step 1: Create a fresh Python environment to avoid runtime conflicts and incompatibilities.
conda create -n fmspeech python=3.12
conda activate fmspeech
Step 2: Install required packages
# Install vLLM (Specifically version 0.13.0)
pip install vllm==0.13.0
# Note: If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1,
# please use "pip install -e . -v" to build vLLM from source.
# Install Transformers and Accelerate
pip install transformers==4.57.3
pip install accelerate
# Install Qwen Omni utilities and Flash Attention
pip install qwen-omni-utils -U
pip install -U flash-attn --no-build-isolation
Step 3: Run Inference Prepare a sample audio file and run the inference script to generate the 14-dimension JSON output.
python infer.py
(See infer.py in our repository for detailed loading and inference examples).