DistilBERT MCQ Correctness Classifier

Author: Anurag Singh
Framework: Hugging Face Transformers + PyTorch
Base Model: distilbert-base-uncased
Task: Binary classification for multiple-choice question answering

Model Description

This model is a fine-tuned DistilBERT model developed for the Smart MCQ Solver Challenge.

The original task contains a question with five answer options (A-E). The problem was converted into a binary classification task by creating one training example for each (question, option) pair.

• Label 1: The option is correct.
• Label 0: The option is incorrect.

During inference, the model predicts a probability for each option, and the option with the highest probability is selected as the final answer.

Problem Formulation

Original MCQ

Question: What is the capital of France?

• A. Berlin
• B. Madrid
• C. Paris
• D. Rome
• E. Lisbon

Converted Binary Samples

Model Architecture

This model is based on DistilBERT, a compressed version of BERT that retains most of BERT's language understanding capability while being significantly smaller and faster.

Architecture

                      text Question + Option Text
                                  ↓
                          DistilBERT Tokenizer
                                  ↓
                              Input IDs
                                  ↓
                      DistilBERT Encoder (6 Transformer Layers)
                                  ↓
                      Contextual Representation
                                  ↓
                      Classification Head (Linear Layer)
                                  ↓
                      Binary Correctness Score

Model Details

Input Representation

Each training sample is constructed as:

Question: <question> Option: <option>

The text is tokenized using the DistilBERT WordPiece tokenizer and converted into:

• input_ids
• attention_mask

Classification

The contextual representation produced by DistilBERT is passed to a linear classification head that outputs logits for two classes:

• Class 0: Incorrect option
• Class 1: Correct option

During inference, the probability of class 1 is used as the correctness score for each option. The option with the highest score among A-E is selected as the final answer.

distilbert_output

This model is a fine-tuned version of bert-base-uncased on the private dataset. It achieves the following results on the evaluation set:

• Loss: 0.0985
• Accuracy: 0.986
• Precision: 0.9769
• Recall: 0.9525
• F1: 0.9646

Intended uses & limitations

Intended Uses

• Multiple-choice question answering
• Educational NLP applications
• Research on transformer-based text classification
• Binary text classification
• Demonstration of parameter-efficient NLP models

Limitations

• The model is trained only on the project dataset and may not generalize well to unrelated domains.
• Performance depends on the quality and wording of the question and options.
• The model evaluates each question-option pair independently and does not explicitly model relationships between all options simultaneously.
• It is intended for educational and research purposes and should not be used in high-stakes decision-making systems without further validation.

Training and evaluation data

The model was trained on a multiple-choice question dataset where each question contained five candidate answers (A-E).

The original dataset was converted into a binary classification dataset by generating one training sample for every (question, option) pair.

Training labels:

• 0 → Incorrect option
• 1 → Correct option

Each input was tokenized using the DistilBERT tokenizer with appropriate padding and truncation before training.

Model performance was evaluated using:

• Accuracy
• Precision
• Recall
• F1-score
• Validation Loss

Training procedure

Preprocessing

• Combined each question with one candidate option to create the input text.
• Tokenized the input using the DistilBERT WordPiece tokenizer.
• Applied padding and truncation to a fixed maximum sequence length.
• Converted labels into binary classes.

Fine-tuning

The model was fine-tuned using Hugging Face Transformers with the following configuration:

The trained model predicts the probability that a candidate option is correct. For each MCQ, all options are scored independently, and the option with the highest probability is selected as the final answer.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Training results

The DistilBERT model achieved the best overall performance among the evaluated models in this project, outperforming both the custom BiLSTM and BERT models in terms of validation performance and Kaggle leaderboard score. The model was selected as the final submission because it provided the best balance between prediction accuracy, computational efficiency, and inference speed.

Framework versions

• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 4.8.5
• Tokenizers 0.22.2