ML-LaySum Qwen3-1.7B SFT
A full-parameter supervised fine-tuning of Qwen3-1.7B for English lay summarization of machine-learning research. It takes the abstract, introduction, and conclusion of a paper and generates a summary for a non-specialist reader. Generation uses Qwen3's non-thinking mode.
Training
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen3-1.7B (chat checkpoint) |
| Base revision | 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e |
| Dataset | ML-LaySum; ICML 2025 and 2026 papers and lay summaries |
| Dataset revision | a983411ae5100d8805f2852e10ebd43647001586 |
| Training / validation / held-out test | 6,911 / 879 / 852 |
| Source | Abstract + introduction + conclusion, where available |
| Target | Reference lay summary |
| Objective | Cross-entropy on summary tokens and the turn-ending token; prompt tokens masked |
| Training | Full-parameter SFT, 3 epochs, 1,296 optimizer steps |
| Learning rate | 2e-5; cosine schedule; 3% warmup |
| Optimizer | AdamW; weight decay 0 |
| Effective batch size | 16 (8 GPUs × 1 example × 2 accumulation steps) |
| Precision | FP32 weights; BF16 mixed-precision computation |
| Hardware | 8 AMD MI210 GPUs |
| Context | Maximum configured length 32,768 tokens; no truncation or packing |
| Seed | 42 |
| Selection | Last checkpoint, step 1,296; not selected by best validation loss |
Training resumed at step 1,000 after a storage-quota interruption, retaining optimizer, scheduler and random states. Test data were not used for training or checkpoint selection. Missing conclusion sections remain missing; no synthetic conclusions were inserted.
Usage
Install a Qwen3-compatible Transformers version (training used 4.57.6), PyTorch and Accelerate.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sandeep123/ML-LaySum-Qwen3-1.7B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
paper_text = "Abstract: ...\n\nIntroduction: ...\n\nConclusion: ..."
instruction = (
"Write a lay summary of the following machine learning research for a "
"non-specialist reader. Explain the problem, main approach, and findings in "
"clear language, faithfully using only the supplied paper text. Return only "
"the summary.\n\nPaper text:\n"
)
text = tokenizer.apply_chat_template(
[{"role": "user", "content": instruction + paper_text}],
tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
The decoding settings above are an example, not a validated optimal configuration. Uploaded weights are FP32 (approximately 6.9 GB); loading in BF16 reduces weight memory to approximately 3.4 GB, excluding activations and the generation cache.
Evaluation and limitations
No held-out generation benchmark or human evaluation has yet been completed. This is an SFT baseline, with no claim of state-of-the-art performance. Outputs can omit qualifications, misstate findings or introduce unsupported explanations. Check summaries against the paper before publication. Validation cross-entropy is a training diagnostic, not a measure of lay-reader comprehension. See training_summary.json for validation losses.
Files and license
This repository contains inference weights, tokenizer files and sanitized training metadata. Optimizer states, account credentials, raw training records and private cluster paths are not uploaded. Model weights are released under Apache 2.0, following the base model license. Dataset and source-paper terms are separate; see the dataset card. Dataset access may require permission independently of this public model.