RoBERTo-v2 :: bs / hr / sr Language Identifier

A new and improved version of absltnull/RoBERTo. Memory footprint halved, accuracy shot up by 20 points, and a major shortcut/problem with the previous model solved.

Architecture

Note: Recommended to use with Latin text. Cyrillic is still a battle ground.

Problem

RoBERTo, while impressively capable in differentiating Bosnian, Croatian and Serbian in Wikipedia text, has a massive flaw in its training data.

While testing the model on non-Cyrillic Serbian and Cyrillic Bosnian examples, I've noticed a fatal bias: Cyrillic, whether it be Bosnian or Serbian, is always predicted as Serbian with ~99.97% (0.9997) or even 100% (1.00) confidence.

Looking at the training data for the original model, almost all of the Serbian Wikipedia pages were written entirely in Cyrillic, while none of the Bosnian pages got such treatment, presenting the model with a massive bias: "Cyrillic is always Serbian."

Further testing the model, it has revealed even more Serbian-related bias. Since almost every Serbian example it has seen was in Cyrillic, the model has learned to associate Latin with Bosnian and Croatian, completely ruling Serbian out and entirely tying it to Cyrillic script. It has learned a clear, consistent, yet entirely wrong signal: "Latin is always NOT Serbian."

That cannot be left as-is. Not as long as I'm alive.

Solution

After 4 unsuccessful attempts of trying to teach the model the correct path, "Cyrillc is not Serbian, it can also be Bosnian. Serbian can also be in Latin", while preserving its original intelligence, I decided to think outside of the box and look at the problem in another way:

"The model likes Cyrillic? Snatch it away entirely and force-feed it Latin through its nose."

I've made an entirely new, Latin-only dataset by converting the entirety of the previous dataset to Latin and salting it with new labeled data from Twitter and SETimes. The dataset taught the model some informal language (Twitter) and gave it new formal sources (SETimes) while still keeping its original dominant field (Wikipedia).

The result? Performance shot up by 20 percentage points of accuracy on Latin-only evaluation.

(While I was at it, I also halved the memory it uses to load and run, and it only takes up ~500 MB on disk. You're welcome.)

Evaluation

Now comes the Latin-only evaluation I was talking about. I've compared RoBERTo vs RoBERTo-v2 vs FastText vs CLD2.

Take a look:

As seen in the chart:

• RoBERTo (acc 0.663, f1 0.615): As already mentioned, when you strip away its "Serbian is Cyrillic and Cyrillic only" defense, the model entirely collapses on actual LID benchmarks, sitting on only 66% accuracy, which basically brings it down by 25 points from its original ~91% accuracy it had on the biased evaluation data.
• RoBERTO-v2 / roberto-finetuned (acc 0.867, f1 0.867): The new, Latin-only replacement shows actually promising results with ~87% accuracy, meaning it isn't guessing the language, but failing on ambiguous labeling cases, as explained in the next section.
• FastText (acc 0.551, f1 0.451): Unreliable for real-world HBS labeling usecases, as it's worse on Latin-only than it was on the biased Cyrillic data. Only 5 points more accurate than a coinflip.
• CLD2 (acc 0.442, f1 0.456): Genuinely the worst option you could pick. Statistically, a coinflip would be more reliable at labeling the languages than this model. Don't use this.

The Confusions

Just like in the previous release, this model also fails at differentiating short, ambiguous, or unrelated text (languages outside of HBS, emoji/symbol spam, etc).

Looking at the matrix...

...we can see:

• Out of 2,950 Bosnian examples, it was mislabeled as Croatian 292 times, and mislabeled as Serbian 139 times.
• Out of 3060 Croatian examples, it was mislabeled as Bosnian 321 times, and mislabeled as Serbian 58 times.
• Out of 2987 Serbian examples, it was mislabeled as Bosnian 322 times, and mislabeled as Croatian 68 times.

So we can conclude:

• The model is very good at differentiating Croatian from Serbian (negligable confusion amounts).
• It is more likely to confuse Serbian with Bosnian than vice versa (322 > 139).
• It is more likely to confuse Croatian with Bosnian than vice versa (321 > 292).

And looking at this report...

==================================================
RoBERTo
==================================================
              precision    recall  f1-score   support

          bs       0.54      0.89      0.67      2950
          hr       0.77      0.88      0.82      3060
          sr       0.98      0.21      0.35      2987


==================================================
roberto-finetuned
==================================================
              precision    recall  f1-score   support

          bs       0.80      0.85      0.82      2950
          hr       0.88      0.88      0.88      3060
          sr       0.93      0.87      0.90      2987

...when the model says that something is Bosnian, it actually is Bosnian 80% of the time, compared to the original's ~54%, and it catches ~85% of such cases.
When it says that something is Croatian, it actually is Croatian 88% of the time, compared to the original's 77%, and it catches 88% of such cases.
And when it says that something is Serbian, it actually is Serbian 93% of the time, compared to the original's misleading 98%, and it catches 87% of such cases, whereas the original misses 71% of actual Serbian cases.

How to use

Again, this is still based on XLM-RoBERTa-Base, so the inference is as easy as it was in the last release, only this time the model takes up half as much memory on disk and in RAM because of the new BF16 weights.

# RoBERTo — sr/bs/hr language identification

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

MODEL_NAME = "absltnull/RoBERTo-v2"

# labels (sr/bs/hr) are already baked into the model's config
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)

classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
    device=0 if torch.cuda.is_available() else -1,
)

# Only use Latin. For the love of God.
texts = [
    "Ove sedmice moram da kupim bijeli hljeb.", # bs
    "Ove nedelje moram da kupim beli hleb.", # sr
    "Ovog tjedna moram da kupim bijeli kruh.", # hr
]

# simple usage: top predicted label
predictions = classifier(texts)
for text, pred in zip(texts, predictions):
    print(f"[{pred['label']}] ({pred['score']:.2%})  {text}")

TL;DR

Also, if I haven't made it obvious yet... don't use this with Cyrillic. The model was trained to be good at Latin. That's about it. Cya.