https://huggingface.co/laion/larger_clap_music_and_speech with ONNX weights to be compatible with Transformers.js.
Usage (Transformers.js)
If you haven't already, you can install the Transformers.js JavaScript library from NPM using:
npm i @xenova/transformers
Example: Perform zero-shot audio classification with Xenova/larger_clap_music_and_speech.
import { pipeline } from '@xenova/transformers';
const classifier = await pipeline('zero-shot-audio-classification', 'Xenova/larger_clap_music_and_speech');
const audio = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav';
const candidate_labels = ['mlk speech', 'upbeat pop song'];
const scores = await classifier(audio, candidate_labels);
// [
// { score: 0.9970206022262573, label: 'mlk speech' },
// { score: 0.002979411045089364, label: 'upbeat pop song' }
// ]
Example: Compute text embeddings with ClapTextModelWithProjection.
import { AutoTokenizer, ClapTextModelWithProjection } from '@xenova/transformers';
// Load tokenizer and text model
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/larger_clap_music_and_speech');
const text_model = await ClapTextModelWithProjection.from_pretrained('Xenova/larger_clap_music_and_speech');
// Run tokenization
const texts = ['an upbeat pop song', 'a passionate speech'];
const text_inputs = tokenizer(texts, { padding: true, truncation: true });
// Compute embeddings
const { text_embeds } = await text_model(text_inputs);
// Tensor {
// dims: [ 2, 512 ],
// type: 'float32',
// data: Float32Array(1024) [ ... ],
// size: 1024
// }
Example: Compute audio embeddings with ClapAudioModelWithProjection.
import { AutoProcessor, ClapAudioModelWithProjection, read_audio } from '@xenova/transformers';
// Load processor and audio model
const processor = await AutoProcessor.from_pretrained('Xenova/larger_clap_music_and_speech');
const audio_model = await ClapAudioModelWithProjection.from_pretrained('Xenova/larger_clap_music_and_speech');
// Read audio and run processor
const audio = await read_audio('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav');
const audio_inputs = await processor(audio);
// Compute embeddings
const { audio_embeds } = await audio_model(audio_inputs);
// Tensor {
// dims: [ 1, 512 ],
// type: 'float32',
// data: Float32Array(512) [ ... ],
// size: 512
// }
Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).