Galactica-125M — SciTLDR

facebook/galactica-125m fine-tuned on allenai/scitldr for scientific paper TL;DR generation (extreme summarization of scientific text).

Model details

Training data

allenai/scitldr, configuration Abstract — the source field contains sentences of the paper abstract.

The Hub repository is script-based, so the data is read from its automatic Parquet conversion (revision="refs/convert/parquet"); content and splits are identical. The official test split was not modified.

The model reads the paper abstract, not the full paper text. Source sentences are joined with a space and head-truncated to 512 tokens; 0.10 % of training examples and 0.00 % of test examples are affected.

Input format

The model is trained as a plain causal LM — no chat or instruction template:

Paper:
<source paper text>

TL;DR:
<target TL;DR></s>

Prompt tokens (Paper: prefix, source text, TL;DR: delimiter) are set to -100 in labels, so the loss is computed only over the TL;DR tokens and the terminating </s>. Galactica's tokenizer emits no BOS token and none is added.

Galactica's published tokenizer has an empty special_tokens_map.json; the <pad> / </s> / <s> roles (ids 1 / 2 / 0) are bound explicitly at load time. The base checkpoint's weights are fp16, which produces NaN on the first AdamW step — training loads the model in fp32 and uses mixed precision.

Training procedure

Evaluation

Generation-based evaluation on the untouched official test split (618 examples) with deterministic decoding:

max_new_tokens = 64
min_new_tokens = 5
no_repeat_ngram_size = 3
repetition_penalty = 1.0
do_sample = False
num_beams = 4
length_penalty = 1.0

ROUGE is computed multi-reference (best reference per example) with Porter stemming, the standard SciTLDR protocol.

Mean generated length: 19.5 words / 22.0 tokens.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "thealper2/galactica-125m-scitldr"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.float32).eval()

# Galactica binds no special-token roles by default.
tokenizer.pad_token = tokenizer.pad_token or "<pad>"
tokenizer.eos_token = tokenizer.eos_token or "</s>"

abstract = "..."  # paper abstract
prompt = f"Paper:\n{abstract}\n\nTL;DR:\n"

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
output = model.generate(
    **inputs,
    max_new_tokens=64,
    min_new_tokens=5,
    num_beams=4,
    no_repeat_ngram_size=3,
    do_sample=False,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
# Slice off the prompt; keep only the generated TL;DR.
tldr = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True
).strip()
print(tldr)

Match the training prompt exactly (Paper:\n...\n\nTL;DR:\n), pass add_special_tokens=False, and use left padding for batched generation.

Limitations

• 125M parameters: limited capacity to identify a paper's actual contribution.
• Rare scientific terms, novel method names, notation and numeric results are frequently dropped or altered.
• Input is the abstract, head-truncated at 512 tokens; contributions stated only in the introduction, results or conclusion are not visible to the model.
• The model can state contributions, datasets, baselines or numbers that the source does not support. Outputs are machine-generated text, not verified scientific claims; no factual verification is performed.
• Small decoder-only models repeat; no_repeat_ngram_size mitigates but does not eliminate this.
• ROUGE measures n-gram overlap, not factual correctness.
• SciTLDR is almost entirely machine-learning / computer-science papers; behaviour in other fields is untested.
• Research artifact only. Not a decision-support tool, and not for medical, clinical or other safety-critical use.

License

Inherits the non-commercial cc-by-nc-4.0 license of the facebook/galactica-125m base model.