DistilBERT New Jersey passage classifier — 24-679 Fall 2026
Given a short passage about life in the US Northeast, predicts whether it describes New Jersey (1) or New York / Pennsylvania (0).
Purpose
Homework 2 (fine-tuning a small text model) for 24-679 Designing with AI. Coursework only.
Data
yennik16/ny-pa-nj-text, written by a classmate: 100 passages of
about 200 characters on food, law, transport, geography and culture, each about one state.
| Split | Rows | Original passages | NJ / other |
|---|---|---|---|
| train | 1070 | 70 | 536 / 534 |
| validation | 15 | 15 | 7 / 8 |
| test | 15 | 15 | 8 / 7 |
Published splits used unchanged. No passage crosses splits; validation and test are originals only (both checked in the notebook). Training augmentation (by the dataset author) is character noise, word dropout, word swaps and sentence shuffling at four strengths.
Preprocessing
Input is the text column only. The state column is the label at finer grain and is excluded.
Tokenization is DistilBERT's uncased WordPiece, truncated at 128 tokens (passages are far shorter).
No extra augmentation during training.
Training
AutoGluon Multimodal 1.6.1, distilbert-base-uncased, seed 24679,
5 epochs, batch size 8, weight decay 0.01, layer-wise learning-rate decay, validation after
each epoch, best single checkpoint kept.
Three fine-tuning modes were compared on validation:
| Mode | LR | Trainable | Val accuracy | Val log loss | Fit (s) |
|---|---|---|---|---|---|
bit_fit | 0.001 | 0.08% | 73.3% | 0.6429 | 28 |
full | 2e-05 | 100.00% | 66.7% | 1.7988 | 45 |
norm_fit | 0.0001 | 0.09% | 53.3% | 0.6686 | 22 |
Selected: bit_fit (learning rate 0.001; 52,994 of
66,364,418 parameters trainable), by validation accuracy with log loss as tiebreak.
Results (test, 15 original passages)
| Metric | DistilBERT | Majority class |
|---|---|---|
| Accuracy | 53.3% | 53.3% |
| Balanced accuracy | 51.8% | 50.0% |
| Weighted F1 | 0.507 | 0.371 |
| Precision (NJ) | 0.545 | 0.533 |
| Recall (NJ) | 0.750 | 1.000 |
95% exact (Clopper-Pearson) interval on test accuracy: 26.6% – 78.7%. One error moves accuracy by 6.7 points.
Error analysis
Misclassified holdout passages (validation and test):
- (validation, NJ) predicted NY or PA at P(NJ) = 0.16: "The most popular gas stations are Wawa and QuickChek. Both of these have attached convenience stores that sell fresh coffee and let you orde..."
- (validation, NJ) predicted NY or PA at P(NJ) = 0.25: "The military has strong roots in the state with George Washingtons former home being located in Morristown. The state was also home to many ..."
- (validation, PA) predicted New Jersey at P(NJ) = 0.93: "The middle of the state is very rural while the west and east are much more urban. As a result, either side tends to lean more liberal while..."
- (validation, PA) predicted New Jersey at P(NJ) = 0.79: "The state has an abundance of junkyards in the east, with many acting as U-Pull-its. Harrys is one of the largest, with three locations and ..."
- (test, NJ) predicted NY or PA at P(NJ) = 0.24: "It has a high cost of living and higher property taxes than most of the country. The average salaries also tend to be higher as a result. It..."
- (test, NJ) predicted NY or PA at P(NJ) = 0.38: "The northern part of the state near the George Washington bridge has a very large korean population. As a result, it has one of the largest ..."
- (test, PA) predicted New Jersey at P(NJ) = 0.85: "The property taxes are much lower than NJ, leading to many opting to live in pennsylvania but work in new jersey. It is not uncommon for peo..."
- (test, PA) predicted New Jersey at P(NJ) = 0.65: "Sheetz is the most common gas station/convenience store rest stop and has full kitchens that you can order food at. Get go is also very comm..."
- (test, PA) predicted New Jersey at P(NJ) = 0.99: "Automotive inspections must be done yearly and cover both emissions and safety. There is a separate sticker displayed on windshields for eac..."
- (test, PA) predicted New Jersey at P(NJ) = 0.97: "Open carry of a firearm is legal throughout the state except for in philadelphia where a license is required. Concealed carry is legal throu..."
- (test, NY) predicted New Jersey at P(NJ) = 0.95: "Beer, cider, and seltzers can be bought in most convenience stores, drug stores, bodegas, and gas stations while liquor is sold exclusively ..."
Why. Accuracy by state on the 30 holdout passages: NJ 73%, NY 86%, PA 25%. Half the PA errors are about cars or law, and their closest training passages are New Jersey ones about driving, so the model learned topic rather than state. For 7 of 11 errors, the most similar training passage carries the other label. Naming a state did not raise accuracy, only confidence. The task needs facts (which chains, laws and places belong to which state) that 70 training passages do not contain.
Limitations and ethics
- Small data. 70 original training passages by one author; 30 holdout passages in total.
- One writer's view. Passages reflect one person's impressions, including stereotypes (e.g. drivers, accents). The model learns those associations too.
- Pooled negative class. NY and PA share label 0, so the model is not a three-way state classifier.
- Named states. Some passages name a state; accuracy on those may overstate skill on passages that do not (see the notebook's breakdown).
- Not suitable for any real decision about people or places.
Compute
Tesla T4; three fits totalling 94 s. Python 3.13.15, PyTorch 2.11.0+cu128.
Usage
from pathlib import Path
import pandas as pd
from huggingface_hub import snapshot_download
from autogluon.multimodal import MultiModalPredictor
path = Path(snapshot_download("ssg1/2026-24679-distilbert-nj-text", allow_patterns=["automm/*"]))
model = MultiModalPredictor.load(str(path / "automm"))
model.predict(pd.DataFrame({"text": ["Full-service gas stations are the law here."]})) # 1 = NJ
AI usage disclosure
Claude helped choose the dataset, draft the notebook from the course reference, and draft this card from values computed in the run. Settings, results and error analysis were checked against notebook output. The passages are a classmate's; none were generated for this model.
Author
Sohum Goel, CMU 24-679, Fall 2026. Dataset by yennik16, CC BY 4.0.