Drafter — WMT16 Translate CSEN (Czech → English)
A domain-specific draft model for speculative decoding, fine-tuned on the wmt16_translate_csen cluster from the Flan dataset.
Model Details
| Parameter | Value |
|---|---|
| Architecture | MistralForCausalLM (156M parameters) |
| Base model | Lite-Mistral-150M-v2-Instruct |
| Target model | TurboSparse-Mistral-Instruct (7B, BambooForCausalLM) |
| Domain | WMT16 Translation: Czech → English |
| Training cluster | wmt16_translate_csen_10templates |
| Precision | bfloat16 |
Training
| Parameter | Value |
|---|---|
| Loss function | 0.5 × CrossEntropy + 0.5 × KL-divergence (T=1.0) |
| Training data | 28,500 synthetic samples (knowledge distillation from target top-10 distributions) |
| Validation data | 1,500 samples (5% held-out split) |
| Epochs | 10 (plateau at ~5) |
| Batch size | 32 |
| Learning rate | 5e-5 (cosine schedule, 3% warmup) |
| Best checkpoint | checkpoint-8010 |
| Final eval_loss | 2.260 |
| Final top1_accuracy | 49.96% |
| GPU | 1× RTX 3090 (24 GB) |
Purpose
This model is part of the Domain-Aware Speculative Decoding research project. The hypothesis is that domain-specific draft models achieve higher acceptance rates in speculative decoding compared to a single general-purpose drafter.
The model is trained via knowledge distillation: the target model's top-10 token distributions are used as soft labels, combined with standard cross-entropy on the greedy (trunk) sequence.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-wmt16-translate-csen")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-wmt16-translate-csen")
Related Models
| Model | Domain | eval_loss | top1_acc |
|---|---|---|---|
| drafter-understanding | Understanding (31 clusters) | 2.100 | 65.00% |
| drafter-text-reformulation | Text Reformulation (11 clusters) | 2.151 | 54.34% |
| drafter-mixed-ut | Mixed U+T (42 clusters) | 2.085 | 59.50% |
| drafter-wmt16-translate-tren | Turkish → English | 1.998 | 54.91% |
| drafter-wmt16-translate-deen | German → English | 2.307 | 52.77% |
| drafter-wmt16-translate-ruen | Russian → English | 2.363 | 50.04% |
| drafter-wmt16-translate-csen | Czech → English | 2.260 | 49.96% |
| drafter-wmt16-translate-fien | Finnish → English | 2.252 | 49.36% |
Dataset
Training data: MikhailRudenko/domain-aware-sd-synthetic