Perceiver IO sentiment analysis model (IMDb)

This model is a Perceiver IO sentiment analysis model fine-tuned on the IMDb dataset. It is a training example of the perceiver-io library.

Model description

The configuration of the encoder is equal to the encoder configuration of krasserm/perceiver-io-mlm-imdb, the generic decoder is configured with a classification output adapter (see building blocks).

Intended use and limitations

The model can be used for sentiment analysis of movie reviews.

Model training

Training details are described here. Encoder weights are initialized with the encoder weights of krasserm/perceiver-io-mlm-imdb decoder weights are randomly initialized. In a first step, only decoder weights are trained, with encoder weights frozen. In a second step, all weights are fine-tuned. Training was done with PyTorch Lightning and the resulting checkpoint was converted to this 🤗 model with a library-specific conversion utility.

Usage examples

To use this model you first need to install the perceiver-io library with extension text.

pip install perceiver-io[text]

Then the model can be used with PyTorch. Either use the model and tokenizer directly

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from perceiver.model.text import classifier  # auto-class registration

repo_id = "krasserm/perceiver-io-txt-clf-imdb"

model = AutoModelForSequenceClassification.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

mini_reviews = [
    "I've seen this movie yesterday and it was really boring.",
    "I can recommend this movie to all fantasy movie lovers.",
]

encoding = tokenizer(mini_reviews, padding=True, return_tensors="pt")
logits = model(**encoding).logits

predictions = logits.argmax(dim=-1)

for r, p in zip(mini_reviews, predictions.numpy()):
    print(f"{r} ({model.config.id2label[p]})")

I've seen this movie yesterday and it was really boring. (NEGATIVE)
I can recommend this movie to all fantasy movie lovers. (POSITIVE)

or use a sentiment-analysis pipeline:

from transformers import pipeline
from perceiver.model.text import classifier  # auto-class registration

repo_id = "krasserm/perceiver-io-txt-clf-imdb"

mini_reviews = [
    "I've seen this movie yesterday and it was really boring.",
    "I can recommend this movie to all fantasy movie lovers.",
]

sentiment_pipeline = pipeline("sentiment-analysis", model=repo_id)
predictions = sentiment_pipeline(mini_reviews)

for r, p in zip(mini_reviews, predictions):
    print(f"{r} ({p['label']})")

I've seen this movie yesterday and it was really boring. (NEGATIVE)
I can recommend this movie to all fantasy movie lovers. (POSITIVE)

Checkpoint conversion

The krasserm/perceiver-io-mlm-imdb model has been created from a training checkpoint with:

from perceiver.model.text.classifier import convert_imdb_classifier_checkpoint

convert_imdb_classifier_checkpoint(
    save_dir="krasserm/perceiver-io-txt-clf-imdb",
    ckpt_url="https://martin-krasser.com/perceiver/logs-0.8.0/txt_clf/version_1/checkpoints/epoch=006-val_loss=0.156.ckpt",
    tokenizer_name="krasserm/perceiver-io-mlm",
    push_to_hub=True,
)