MCQ Scratch Solver — BiLSTM + Transformer
A custom multiple-choice QA model built from scratch (no pretrained backbone) for the Smart MCQ Solver Challenge on Kaggle.
Architecture
Embedding → BiLSTM (bidirectional) → LayerNorm → Transformer Encoder (self-attention) → Mean+Max Pooling → Linear classifier
Each (prompt, option) pair is scored independently; the 5 option scores per question form the final logits, ranked to produce a top-3 prediction (MAP@3 metric).
Files
This repo stores weights as a plain PyTorch state dict (pytorch_model.bin) rather
than safetensors, due to shared-storage tensors in the LSTM layer that are
incompatible with safetensors' serialization checks. Config (architecture
hyperparameters) is stored separately in config.json.
Training
- Tokenizer:
bert-base-uncased(WordPiece, used for input encoding only) - Embedding dim: 128, LSTM hidden: 128, Attention heads: 4, Transformer layers: 2
- Optimizer: AdamW, lr=1e-3, weight decay 0.01
- Loss: CrossEntropyLoss with label smoothing (0.1)
- Epochs: 5, batch size: 16
- Validation MAP@3: 1.0
Usage
import torch, json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import torch.nn as nn
class HybridQAModel(nn.Module):
def __init__(self, vocab_size, embed_dim, lstm_hidden, num_heads, num_layers):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
self.lstm = nn.LSTM(embed_dim, lstm_hidden, batch_first=True, bidirectional=True)
self.layer_norm = nn.LayerNorm(lstm_hidden * 2)
encoder_layer = nn.TransformerEncoderLayer(d_model=lstm_hidden * 2, nhead=num_heads, batch_first=True)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
self.classifier = nn.Linear(lstm_hidden * 4, 1)
def forward(self, input_ids):
b, n, s = input_ids.shape
flat = input_ids.view(b * n, s)
mask = flat == 0
emb = self.embedding(flat)
out, _ = self.lstm(emb)
out = self.layer_norm(out)
enc = self.transformer(out, src_key_padding_mask=mask)
pooled = torch.cat([enc.mean(1), enc.max(1)[0]], dim=1)
return self.classifier(pooled).view(b, n)
repo_id = "pranay-aggarwal/mcq-scratch-lstm-transformer"
config_path = hf_hub_download(repo_id=repo_id, filename="config.json")
weights_path = hf_hub_download(repo_id=repo_id, filename="pytorch_model.bin")
with open(config_path) as f:
config = json.load(f)
model = HybridQAModel(**config)
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()
tokenizer = AutoTokenizer.from_pretrained(repo_id)
Live demo
Author
Built as part of BSDA2001P: Introduction to DL and GenAI (project coursework).