WHISPER-SMALL - GGUF Quantized Models

Quantized versions of openai/whisper-small in GGUF format.

Directory Structure

small/
├── whisper-small-q*.gguf       # Candle-compatible GGUF models (root)
├── config.json                # Model configuration for Candle
├── tokenizer.json             # Tokenizer for Candle
└── whisper.cpp/               # whisper.cpp-compatible models
    └── whisper-small-q*.gguf

Format Compatibility

• Root directory (whisper-small-*.gguf): Use with Candle (Rust ML framework)

  • Tensor names include model. prefix (e.g., model.encoder.conv1.weight)
  • Compatible with Neurolang application
  • Requires config-small.json and tokenizer-small.json

• whisper.cpp/ directory: Use with whisper.cpp (C++ implementation)

  • Tensor names without model. prefix (e.g., encoder.conv1.weight)
  • Compatible with whisper.cpp CLI tools
  • Both directories contain .gguf files, not .bin files

Available Formats

Usage

With Candle (Rust)

For this model, you need to modify the example code in candle. To try whisper in candle faster and easier, it's better to use the tiny model → https://huggingface.co/oxide-lab/whisper-tiny-GGUF

Command line example:

# Run Candle Whisper with local quantized model
cargo run --example whisper --release -- \
  --features symphonia \
  --quantized \
  --model small \
  --model-id oxide-lab/whisper-small-GGUF \

With whisper.cpp (C++)

# Use models from whisper.cpp/ subdirectory
./whisper.cpp/build/bin/whisper-cli \
  --model models/openai/small/whisper.cpp/whisper-small-q4_k.gguf \
  --file audio.wav

Recommended Format

For most use cases, we recommend q4_k format as it provides the best balance of:

• Size reduction (~65% smaller)
• Quality (minimal degradation)
• Speed (faster inference than higher quantizations)

Quantization Details

• Source Model: openai/whisper-small
• Quantization Methods:
  • Candle GGUF (root directory): Python-based. Directly PyTorch → GGUF
    • Adds model. prefix to tensor names for Candle compatibility
  • whisper.cpp GGML (whisper.cpp/ subdirectory): whisper-quantize tool
    • Uses original tensor names without prefix
• Format: GGUF (GGML Universal Format) for both directories
• Total Formats: 10 quantization levels (q2_k through q8_0)

License

Same as the original Whisper model (MIT License).

Citation

@misc{radford2022whisper,
  doi = {10.48550/ARXIV.2212.04356},
  url = {https://arxiv.org/abs/2212.04356},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}