drivesignal-distilbert
Fine-tuned distilbert-base-uncased that classifies free-text descriptions of
autonomous-vehicle disengagement events into a 10-category safety scenario
taxonomy. Trained on real California DMV Autonomous Vehicle Disengagement
Reports (2022–2024).
Part of DriveSignal, a project comparing a rule-based heuristic, this fine-tuned DistilBERT classifier, and a LoRA-fine-tuned Qwen3-1.7B on the same task.
Taxonomy
Perception Failure | Prediction Failure | Lane Keeping | Braking Behavior |
Unwanted Maneuver | Construction/Environment | Precautionary |
System/Hardware Fault | Localization/Mapping | Other
Training data
- 14,800 disengagement event descriptions from the CA DMV 2022–2024 reports (Waymo and other manufacturers).
- Weak-labeled: training labels come from a keyword/regex heuristic
classifier, not human annotation — a standard weak-supervision pattern for
bootstrapping labels when ground truth doesn't exist. See training script:
src/finetune_distilbert.py.
Evaluation
Evaluated against a 250-row hand-labeled gold set (not seen during training, and not derived from the heuristic labels):
| Model | Accuracy (n=250) |
|---|---|
| Heuristic baseline | 80.0% |
| DistilBERT (this model) | 80.0% |
| Qwen3-1.7B LoRA | 72.8% |
Full breakdown: src/eval_distilbert_gold.py.
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="jahnavidanda02/drivesignal-distilbert")
clf("Vehicle disengaged after hesitating at an unprotected left turn due to "
"misjudging the trajectory of an oncoming vehicle.")
Limitations
- Trained on weak (heuristic-derived) labels for the bulk of the data, so it inherits gaps in the heuristic's keyword coverage (e.g. rare categories like Construction/Environment and Precautionary are underrepresented).
- Trained only on CA DMV disengagement report language; may not generalize to differently phrased AV incident text.