https://huggingface.co/microsoft/Florence-2-large-ft with ONNX weights to be compatible with Transformers.js.
Usage (Transformers.js)
If you haven't already, you can install the Transformers.js JavaScript library from NPM using:
npm i @huggingface/transformers
Example: Perform image captioning with onnx-community/Florence-2-large-ft.
import {
Florence2ForConditionalGeneration,
AutoProcessor,
load_image,
} from '@huggingface/transformers';
// Load model, processor, and tokenizer
const model_id = 'onnx-community/Florence-2-large-ft';
const model = await Florence2ForConditionalGeneration.from_pretrained(model_id, {
dtype: {
embed_tokens: 'fp16', // or 'fp32'
vision_encoder: 'fp16', // or 'fp32'
encoder_model: 'q4',
decoder_model_merged: 'q4',
},
});
const processor = await AutoProcessor.from_pretrained(model_id);
// Load image and prepare vision inputs
const url = 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg';
const image = await load_image(url);
// Specify task and prepare text inputs
const task = '<MORE_DETAILED_CAPTION>';
const prompts = processor.construct_prompts(task);
// Pre-process the image and text inputs
const inputs = await processor(image, prompts);
// Generate text
const generated_ids = await model.generate({
...inputs,
max_new_tokens: 256,
});
// Decode generated text
const generated_text = processor.batch_decode(generated_ids, { skip_special_tokens: false })[0];
// Post-process the generated text
const result = processor.post_process_generation(generated_text, task, image.size);
console.log(result);
// { '<MORE_DETAILED_CAPTION>': 'A car is parked on the street. The car is a light green color. The doors on the building are brown. The building is a yellow color. There are two doors on both sides of the car. The wheels on the car are very shiny. The ground is made of bricks. The sky is blue. The sun is shining on the top of the building.' }
We also released an online demo, which you can try yourself: https://huggingface.co/spaces/Xenova/florence2-webgpu
Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).