gemma-3-270m-bhaskar-finetune
google/gemma-3-270m-it fully fine-tuned on a synthetic
legal/financial instruction set, to answer questions grounded in a passage supplied in
the prompt.
Results
Held-out split, loss over assistant tokens only:
| value | |
|---|---|
base gemma-3-270m-it | 2.0595 |
| this model | 1.2213 (ppl 3.391) |
| best epoch | 2 of 3 |
Trained on L4 x1 in 37 minutes (10,520 tok/s, peak 20.29 GB).
Versus a 125M model trained from scratch on the same data
The same dataset was used to fine-tune nagbhaskar55/slm125mlive-base, a 125M model pretrained from scratch on 2.04B tokens. Per-token perplexity is not comparable across a 16k and a 262k vocabulary, so the like-for-like measure is bits per character of the reference answers:
| model | params | pretrain tokens | bits/char |
|---|---|---|---|
| slm125mlive-base + SFT | 125M | 2.04B | 0.4915 |
| this model | 268M | ~6T | 0.3964 |
A 19% reduction. Both were fine-tuned on identical data, so the gap reflects pretraining scale rather than fine-tuning.
Training data
7,960 curated synthetic instruction pairs generated from US case law, SEC
filings and educational web text, with gemini-3.6-flash as teacher and
gemini-3.1-flash-lite as an LLM judge (grounding / correctness / instruction-following,
each >= 4 of 5), then exact + n-gram + embedding deduplication and 13-gram
decontamination against CaseHOLD.
- Tasks: {'summarization': 1600, 'extraction': 1600, 'grounded_qa': 3200, 'rewriting': 1560}
- Sources: {'sec': 3186, 'case-law': 3158, 'fineweb-edu': 1616}
- Refusals: 423 items answer exactly
"Not stated in the context." - Re-tokenized for Gemma: 7,617 train examples, 4,896,142 tokens, 8.7% carrying loss. 105 examples exceeded 1,024 tokens and were dropped rather than truncated.
Prompt format
Gemma's own chat template. It has no separate system channel — a system turn is folded into the first user turn, which is how this model was trained:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nagbhaskar55/gemma-3-270m-bhaskar-finetune")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/gemma-3-270m-bhaskar-finetune")
system = "You are a legal and financial assistant. Use only the provided context."
messages = [{"role": "user", "content": f"{system}\n\n{passage}\n\n{question}"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(
ids, max_new_tokens=200,
# Gemma turns end with <end_of_turn> (106), not <eos> (1). Pass both, or the
# model will not stop.
eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<end_of_turn>")],
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Loss was applied only to the model turn (8.7% of tokens).
Limits
At 270M parameters this reads the passage you give it; it is not a knowledge base. Answers are fluent and usually well-formed, but figures and entities are still sometimes wrong, and it can miss part of a multi-part question. Training data is synthetic, from a single teacher model, so its biases carry over. Not legal or financial advice. Licensed under the Gemma Terms of Use, which carry use restrictions and must be passed downstream.