mmBERT-small-NLI

A multilingual Natural Language Inference (NLI) model fine-tuned from jhu-clsp/mmBERT-small, which supports 1833 languages. This model was fine-tuned on a comprehensive combination of 9 NLI datasets to enable strong NLI and zero-shot classification across a massive range of languages.

What is this model?

The base model jhu-clsp/mmBERT-small was pre-trained by Johns Hopkins University on 1833 languages for general language understanding. We fine-tuned it specifically for the Natural Language Inference (NLI) task — teaching it to determine whether a hypothesis is:

• ✅ Entailment — the hypothesis follows from the premise
• ❓ Neutral — the hypothesis may or may not follow
• ❌ Contradiction — the hypothesis contradicts the premise

Training Data

This model was fine-tuned on 9 NLI datasets combining over 1.8 million training examples across multiple languages:

Total training examples: ~2.1 million pairs across 26+ languages

Benchmark Results

Evaluated on standard NLI test sets after training:

Note on ANLI scores: ANLI is intentionally adversarial and designed to fool masked language models. Even large models like RoBERTa-large score ~47% on ANLI. Low ANLI scores are expected for small models.

Comparison with Other NLI Models

Key advantage: This is the only NLI model covering 1833 languages, compared to the next best multilingual NLI model (mDeBERTa) covering only 100 languages.

How to Use

Zero-Shot Classification

from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="BalaRajesh1/mmbert-small-nli"
)

# English
result = classifier(
    "The Federal Reserve raised interest rates today.",
    candidate_labels=["economics", "politics", "sports"]
)
print(result)

# Hindi
result = classifier(
    "सरकार ने नई शिक्षा नीति की घोषणा की।",
    candidate_labels=["education", "politics", "sports"]
)
print(result)

# Arabic
result = classifier(
    "أعلنت الحكومة عن خطة جديدة للطاقة المتجددة.",
    candidate_labels=["environment", "politics", "technology"]
)
print(result)

Direct NLI

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "BalaRajesh1/mmbert-small-nli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

premise = "The cat is sitting on the mat."
hypothesis = "There is an animal on the mat."

inputs = tokenizer(premise, hypothesis, return_tensors="pt", truncation=True)
with torch.no_grad():
    logits = model(**inputs).logits

probs = torch.softmax(logits, dim=-1)
labels = ["entailment", "neutral", "contradiction"]
for label, prob in zip(labels, probs[0]):
    print(f"{label}: {prob:.3f}")

Training Details

Training was stopped early at ~19% of maximum steps because the model converged and validation F1 stopped improving — this is expected behavior, not an error.

Label Mapping

Limitations

• ANLI performance is low (~40%) — expected for small models on adversarial data
• Performance may vary across the 1833 languages depending on how well represented they are in the base mmBERT pre-training
• Max sequence length of 128 tokens — very long premise+hypothesis pairs will be truncated

Citation

If you use this model, please cite the original mmBERT paper:

@misc{mmbert2021,
  title={mmBERT: Multilingual BERT for 1000+ Languages},
  author={Johns Hopkins University CLSP},
  year={2021}
}