mBERT: CM code-mixed offensive language

mBERT fine-tuned for binary classification of Hindi–English code-mixed text on CM code-mixed offensive language. Part of a comparative study of dataset transfer, training mixtures and seed variation.

Research code · Results

• Labels: 0 = NOT_OFFENSIVE, 1 = OFFENSIVE.
• Revision: seed 42; main defaults to seed 42. Available: seed-7, seed-13, seed-42.
• Training sources: CM. No dataset text is included.

Evaluation

Evaluation splits also guided checkpoint selection; the recorded internal scores are not untouched final-test estimates. Cross-dataset results and label definitions should be interpreted separately.

Mean Macro F1: 0.7768 ± 0.0187 (sample standard deviation).

Limitations

Use for research, comparison and error analysis. Cross-dataset generalization is limited; label definitions and platforms differ, duplicates exist in CM, and annotation/source uncertainties remain. Identity words, transliteration, quoted abuse and missing conversation context can cause errors. These checkpoints have not been validated for autonomous moderation or decisions about individuals. Single-seed mixed results do not establish seed robustness.

Install packages from requirements.txt. This repository is public and can be loaded without a Hugging Face login.

import re
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "OmTheLast/mbert-hinglish-cm"
revision = "seed-42"  # use "main" for the default seed 42
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModelForSequenceClassification.from_pretrained(repo, revision=revision)
model.eval()

def clean(text):
    text = re.sub(r"http\S+|www\S+|https\S+", " URL ", str(text))
    text = re.sub(r"@\w+", " USER ", text)
    return re.sub(r"\s+", " ", text).strip()

text = clean("Aaj ka din accha tha.")
inputs = tokenizer(text, return_tensors="pt", padding="max_length",
                   truncation=True, max_length=128)
with torch.inference_mode():
    scores = model(**inputs).logits.softmax(dim=-1)[0]
print({model.config.id2label[i]: float(score) for i, score in enumerate(scores)})

Preprocessing preserves case, replaces URLs with URL, handles with USER, and collapses whitespace. Truncation is 128 tokens including special tokens.

See training_metadata.json for this run's settings, row counts, split policy and any reconstructed fields. Runs used two training epochs; the script restores the epoch with the highest evaluation Macro F1. The exported weights may therefore come from an earlier epoch.

CM uses source train+val for training and source test for evaluation, with the same split across seeds.

Evaluation limitation: the training script evaluates each epoch on this evaluation split and uses it to choose the checkpoint. In CM this includes the source test split. The following scores are therefore selection-set results, not an untouched final test estimate. The training seed was supplied to Trainer; exact reproduction of classifier initialization and device behavior is not guaranteed.

Internal / matched evaluation recorded with checkpoints

Across seeds 7, 13, 42: Macro F1 0.7768 ± 0.0187 (sample standard deviation, not a confidence interval). This describes the group, not one checkpoint.

All scores above come directly from the saved eval_metrics.json files. Historical *_hate keys mean the dataset-specific positive class. The 79-row diagnostic probe is excluded from these claims.

Public research checkpoint release. Both base-model cards declare Apache-2.0 (mBERT, MuRIL). A license for these fine-tuned releases has not yet been assigned: training-source terms remain under review, including the unresolved CM and THAR licenses. The base-model license is not a license grant for the datasets. Public access does not assign a new license to the fine-tuned releases or their training data. The upstream Apache-2.0 license is included in BASE_MODEL_LICENSE.txt; see BASE_MODEL_NOTICE.md for attribution.

• Research code, source snapshot 4650c6eb093d422785284656dd6765d36e438522.
• Working paper draft.
• Dataset registry.
• Model registry and failure cases.
• release_metadata.json records original checkpoint identity and SHA-256. Weights are unmodified; label names were added to the exported configuration.
• Seeds 42 for the two Kaggle models have reconstructed metadata, with provenance and uncertainty recorded in their metadata files.

Tools Note

AI tools were used for coding, debugging, and documentation assistance; the research direction, result interpretation, and final claims were reviewed and owned by Om Patnaik.