Origin Task 1 — T5Gemma 2 270M-270M
This is a full-parameter fine-tuned checkpoint of google/t5gemma-2-270m-270m for SciHigh-2026 Task 1. It generates concise research highlights from scientific paper abstracts.
Model Details
- Parameters: 786M trainable
- Framework: Transformers 5.14.1, PyTorch 2.13.0+cu130
- License: Must be used in accordance with upstream Gemma terms.
- Training Data: 10,000 examples (SciHigh Task 1 MixSub-SciHigh split)
- Hyperparameters: 3 epochs, lr
3e-5, batch size 32, max input/target lengths 1024/320, bfloat16, SDPA attention.
Validation Results (1,985 examples)
| Metric | Score |
|---|---|
| ROUGE-1 | 0.3818 |
| ROUGE-2 | 0.1389 |
| ROUGE-L / Lsum | 0.2590 / 0.2591 |
| METEOR | 0.2983 |
| BERTScore F1 | 0.8775 |
Inference
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "uthayamurthy/origin-task1-t5gemma2-270m"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForSeq2SeqLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
attn_implementation="sdpa",
).to("cuda").eval()
abstract = "..."
inputs = tokenizer(abstract, return_tensors="pt", truncation=True, max_length=1024)
inputs = {name: value.to("cuda") for name, value in inputs.items()}
with torch.inference_mode():
generated = model.generate(
**inputs,
max_length=128,
min_length=0,
num_beams=4,
length_penalty=1.0,
no_repeat_ngram_size=3,
early_stopping=True,
)
print(tokenizer.decode(generated[0], skip_special_tokens=True))