ag-news-classifier
A DeBERTa-v3-base model fine-tuned to classify English news text into three topics: Sports, Business, and Sci/Tech.
This is a 3-class model, not the standard 4-class AG News setup. The World
category was removed from both the training and evaluation splits, and the
remaining labels were remapped to 0, 1, 2.
Labels
| id | label |
|---|---|
| 0 | Sports |
| 1 | Business |
| 2 | Sci/Tech |
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="Bubunur/ag-news-classifier")
clf("The stock market rallied after the central bank's decision")
Or directly:
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "Bubunur/ag-news-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
labels = ["Sports", "Business", "Sci/Tech"]
text = "Researchers unveiled a chip that runs models on-device"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
probs = model(**inputs).logits.softmax(-1)[0]
print(labels[probs.argmax()], probs.max().item())
Training data
AG News, filtered to remove
the World class.
| split | examples |
|---|---|
| train | 90,000 |
| test | 5,700 |
Classes are perfectly balanced: 30,000 training and 1,900 test examples each. Each example is a news title concatenated with a short description. Texts are short — truncation at 128 tokens affects almost no examples.
Training procedure
Fine-tuned from microsoft/deberta-v3-base using the HuggingFace Trainer.
| hyperparameter | value |
|---|---|
| epochs | 3 |
| batch size | 16 |
| learning rate | 2e-5 |
| optimizer | AdamW |
| weight decay | 0.01 |
| LR schedule | linear with 6% warmup |
| max sequence length | 128 (dynamic padding) |
| precision | fp16 mixed precision |
| seed | 42 |
| hardware | single GPU (Google Colab) |
| wall-clock time | ~50 min |
Per-epoch results:
| epoch | training loss | validation loss | accuracy | macro F1 |
|---|---|---|---|---|
| 1 | 0.1439 | 0.1429 | 0.9542 | 0.9542 |
| 2 | 0.1645 | 0.1405 | 0.9563 | 0.9563 |
| 3 | 0.1045 | 0.1745 | 0.9574 | 0.9573 |
The epoch-3 checkpoint was selected on accuracy. Note that validation loss rose sharply at epoch 3 while accuracy continued to climb — a typical sign that the model is becoming overconfident on examples it already gets right. The accuracy gain over epoch 2 (+0.11 pp) is small enough to fall within run-to-run noise.
Implementation note: microsoft/deberta-v3-base ships fp16 weights. Loading it
without forcing dtype=torch.float32 raises ValueError: Attempting to unscale FP16 gradients when fp16=True is enabled.
Evaluation
Measured on the 5,700-example AG News test split (World removed).
| metric | value |
|---|---|
| accuracy | 0.9574 |
| macro F1 | 0.9573 |
Because the classes are perfectly balanced, accuracy and macro F1 coincide.
An earlier run of the same architecture under a slightly different configuration reached 0.9560 accuracy, with this per-class breakdown:
| class | precision | recall | F1 | support |
|---|---|---|---|---|
| Sports | 0.99 | 0.99 | 0.99 | 1900 |
| Business | 0.95 | 0.92 | 0.94 | 1900 |
| Sci/Tech | 0.93 | 0.95 | 0.94 | 1900 |
Sports Business Sci/Tech
Sports 1888 6 6
Business 11 1752 137
Sci/Tech 12 79 1809
The overall shape is expected to hold for the current checkpoint, but these per-class figures come from the earlier run and have not been recomputed.
Domain and period. AG News was collected from news wires in the early 2000s. Performance on social media text, long-form articles, non-news prose, or non-English text is untested and expected to be worse. Terminology that emerged after the collection period was never seen during training. The model has not been evaluated on any corpus other than AG News, so its ability to generalize to other news sources is unknown.
Inherited bias. Any bias present in microsoft/deberta-v3-base or in the
AG News corpus carries over. No bias audit was performed.
Citation
@misc{zhang2015character,
title={Character-level Convolutional Networks for Text Classification},
author={Xiang Zhang and Junbo Zhao and Yann LeCun},
year={2015},
eprint={1509.01626},
archivePrefix={arXiv}
}