CARD-Qwen3-4B-Clotho

Encoder-free audio captioning model from the paper "CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning" (IEEE SLT 2026). Code: https://github.com/KarthikKolluriKB/CARD

CARD removes the audio encoder at inference. A 13.2 M-parameter convolutional projector turns a log-Mel spectrogram into about 250 audio tokens, which a Qwen3-4B language model (with LoRA adapters merged into its weights) reads to write the caption. During training a frozen CLAP-HTSAT teacher was distilled into the model by component: its early, perceptual stages into the projector and its later, semantic stages into the LLM. The teacher and all distillation heads are gone at inference.

This repo is the CARD* model fine-tuned (Phase 2) on Clotho. It shares its Phase 1 checkpoint with the AudioCaps model. Related repos:

• KarthikKB1998/CARD-Qwen3-4B-AudioCaps: the same Phase 1 checkpoint fine-tuned on AudioCaps.
• KarthikKB1998/CARD-Qwen3-4B-Phase1: the shared Phase 1 checkpoint (before Phase 2).

CARD training (left) and inference (right) pipelines (paper Fig. 1). A frozen CLAP teacher's early, perceptual stages are distilled into the audio projector and its later, semantic stages into the LLM's LoRA adapters; at inference only the projector and the merged LLM remain.

What is in this repo

You need only llm/ and audio_projector.safetensors to run the model. The adapters are provided so the merged weights can be rebuilt from Qwen/Qwen3-4B (Phase 1 merged first, then Phase 2) and so the two phases can be studied separately.

Quick start

Try it without installing anything: listen to sample clips on the demo page, or caption your own audio in the Colab notebook.

pip install git+https://github.com/KarthikKolluriKB/CARD

On NVIDIA RTX 50-series (Blackwell) GPUs, install the CUDA 12.8 build of PyTorch first (tested on an RTX 5090, where the model gives the same captions as the paper run):

pip install torch==2.7.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/KarthikKolluriKB/CARD

from card import load_card

model = load_card("KarthikKB1998/CARD-Qwen3-4B-Clotho", device="cuda")
print(model.caption("rain_and_thunder.wav"))

To rebuild the merged weights from the base model and the two adapters instead of downloading them:

model = load_card("KarthikKB1998/CARD-Qwen3-4B-Clotho", mode="adapters", device="cuda")

Input contract: any sample rate, mono (stereo is averaged), any length. The front end resamples to 16 kHz, crops or zero-pads to 10 s, then resamples to 48 kHz for the CLAP log-Mel spectrogram, exactly as in training; clips longer than 10 s are truncated (Clotho clips are 15-30 s, so only their first 10 s are captioned). Output is one English sentence. Decoding is beam search (4 beams, 40 tokens) with the prompt "Describe this audio." under the captioning system prompt stored in card_config.json; enable_thinking=False is applied to the Qwen3 chat template.

Requirements: torch==2.7.1, transformers==4.53.1, peft==0.18.1 (only for mode="adapters"), torchaudio, soundfile. Peak GPU memory is 8.4 GB at batch 1 with beam 4 (paper Section IV-B).

Results

1,045 clips of the official Clotho v2.1 validation split, the evaluation set used for the paper, with the reference captions of each clip split on periods (4,680 references; see scripts/data/prepare_clotho.py in the GitHub repo). Beam 4, all values in % (paper Table II). Results are Phase-2-fine-tuned on the Clotho training set, not zero-shot. SLAM-AAC keeps the frozen CLAP encoder at inference; the CARD variants remove it and differ only in where the teacher is distilled.

All numbers are as reported in the paper. CARD*'s gains over no distillation, LLM-only distillation and mismatched projector supervision are statistically significant on both datasets (paired bootstrap over per-clip CIDEr-D, 10,000 resamples).

Training summary

• Backbone: Qwen3-4B, frozen. LoRA r=16, alpha=16, dropout 0, on q/k/v/o/gate/up/down of all 36 blocks. Audio tokens attend to each other bidirectionally during training; text stays causal.
• Teacher (Phase 1 only): laion/clap-htsat-fused, frozen. Its four HTSAT stage outputs (192-d, 384-d, 768-d, 768-d) are the distillation targets. Stages 0-1 supervise the projector through two small MLP heads (mean-pooled cosine loss). Stages 2-3 supervise the LLM's audio-token hidden states from blocks 0-8 and 9-11 through RMSNorm + linear heads (token-level cosine loss, teacher interpolated along time). Loss: L_cap + L_proj + L_llm with both weights 1.
• Phase 1 (1 epoch): WavCaps, Auto-ACD, AudioCaps, Clotho, MACS captions (about 276 K) sampled by size, plus 50 K Infinity-Instruct text-only batches. AdamW, betas (0.9, 0.95), effective batch 64, lr 2e-4 constant with 200 warmup steps, grad clip 1.0.
• Phase 2 (1 epoch): teacher and heads removed, Phase 1 LoRA merged, fresh LoRA (r=16, alpha=16) and the projector fine-tuned on the prompt-expanded Clotho training set (95,975 pairs) with the captioning loss only, lr 2e-5 cosine with 50 warmup steps. Then the Phase 2 LoRA is merged.
• Hardware: 2 x A6000 48 GB.

Model tree: Qwen/Qwen3-4B -> CARD-Qwen3-4B-Phase1 -> CARD-Qwen3-4B-Clotho (this repo).

Intended use and limitations

• Intended for research on encoder-free audio-language models and for captioning short environmental-sound clips in English. Not a speech recogniser: speech is described ("a man speaks"), not transcribed.
• Trained on 10 s windows. Clotho clips are 15-30 s long and only their first 10 s are used.
• The model inherits the biases of its caption datasets (Freesound-sourced Clotho, YouTube-sourced AudioCaps) and of the Qwen3-4B backbone. Captions can be wrong or generic; do not use for safety-critical monitoring.
• Performance is below encoder-kept systems (39.0 vs 27.5 CIDEr-D); the contribution is the encoder-free operating point and the distillation placement result, not state of the art.

License

Weights and code are released under Apache-2.0. The backbone Qwen/Qwen3-4B and the teacher laion/clap-htsat-fused are also Apache-2.0. No training audio is redistributed here.

Citation

@inproceedings{kolluri2026card,
  title     = {{CARD}: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning},
  author    = {Kolluri, Ganesh Pavan Kartikeya Bharadwaj and Zhang, Yuchen and Kampouridis, Michael and Shekhar, Ravi},
  booktitle = {Proceedings of the IEEE Spoken Language Technology Workshop (SLT)},
  year      = {2026}
}

Contact: karthik.kolluri@essex.ac.uk