Gemma3-270M-Pralinka-V1.1
A Czech-language character model. Gemma 3 270M IT, continued-pretrained on Czech and then supervised-fine-tuned to speak as Pralinka — a haflinger mare with opinions who answers questions about horses, stable life, training and trail rides.
At 270M parameters (~540 MB in bf16) it runs comfortably on a CPU or a low-end GPU. It was built as part of a project bridging IT and horsemanship: an accessible way to answer the basic questions children and stable visitors ask about horses.
It is a character, not a knowledge base. Read the Limitations before using it.
Usage
The model expects a chat-formatted message whose content is prefixed with the Czech
instruction Odpověz jako Pralinka: ("Answer as Pralinka:"). This prefix is part of the
training format — keep it verbatim in Czech, the model performs noticeably worse without it.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "RadimSvetlak/Gemma3-270M-Pralinka-V1.1"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
MODEL,
dtype=torch.float32, # float32 is more stable here than fp16
attn_implementation="eager", # recommended for Gemma
)
model.eval()
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
def ask(question: str) -> str:
messages = [{"role": "user", "content": "Odpověz jako Pralinka: " + question}]
ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(
ids,
max_new_tokens=120,
do_sample=False, # greedy is the most reliable setting
repetition_penalty=1.05,
pad_token_id=tokenizer.pad_token_id,
)
return tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip()
print(ask("Co koně jedí?"))
# -> Základem je hlavně seno, tráva a čistá voda.
# ("Mostly hay, grass and clean water.")
Sampling. Greedy decoding (do_sample=False) is the default and the recommended setting.
For livelier output at the cost of reliability, try do_sample=True, temperature=0.7, top_p=0.9, top_k=40. At this model size, sampling measurably increases the rate of nonsense.
Length. max_new_tokens=120 is plenty — the median answer is around 65 characters. The
model is deliberately terse.
Training
Three stages, all full fine-tuning (no LoRA):
| stage | what happened |
|---|---|
| 1 | google/gemma-3-270m-it as the starting point |
| 2 | continued pretraining on Czech → gemma3_270m_it_czech_cpt |
| 3 | SFT on a Pralinka dialogue dataset (pralinka_q_a_v3) |
Stage 3 used trl.SFTTrainer in prompt/completion format with completion_only_loss=True, so
the loss is computed on the answer only, not on the question.
Hyperparameters (stage 3)
| parameter | value |
|---|---|
| epochs | 2 |
| learning rate | 1e-4 |
| weight decay | 0.01 |
| warmup ratio | 0.2 |
| batch size | 2 × grad. accumulation 8 = effective 16 |
| max_length | 256 |
| packing | no |
| precision | bf16 |
| gradient checkpointing | yes |
| attention | eager |
| eval split | 5 % |
| seed | 42 |
Evaluation
An in-house benchmark of 73 questions across six categories, run on CUDA:
| category | questions |
|---|---|
| horse life | 24 |
| identity | 20 |
| trail rides | 10 |
| hard | 8 |
| stable operations | 7 |
| out-of-domain | 4 |
Difficulty: 16 easy, 34 medium, 23 hard. The model answered all 73 questions with no errors and no empty outputs, median 0.78 s per answer, median length 65 characters.
What it does well
Q: Ahoj, kdo jsi? ("Hi, who are you?")
A: Jsem Pralinka. Kobylka s názorem.
("I'm Pralinka. A mare with opinions.")
Q: Co koně jedí? ("What do horses eat?")
A: Základem je hlavně seno, tráva a čistá voda.
("Mostly hay, grass and clean water.")
Q: Jaký je rozdíl mezi anglickým a westernovým sedlem?
("What's the difference between an English and a Western saddle?")
A: Anglické sedlo bývá lehčí a menší, jezdec v něm sedí aktivněji
a používá ho často na jízdárně nebo při sportu.
("An English saddle tends to be lighter and smaller; the rider sits more
actively and it's often used in the arena or for sport.")
Q: Kdo je Venda? ("Who is Venda?")
A: Venda je starší moudrý hřebec a výrazná koňská osobnost.
("Venda is an older, wise stallion and a striking horse personality.")
Where it fails
Outside the horse domain the model does not work and produces nonsense — sometimes down to broken words:
Q: Co je integrál? ("What is an integral?")
A: integrál je výborný program, při které samotné já se s koněm
pracuje ze země nebo se učím v klidu spolu s ostatními.
("an integral is an excellent programme, in which I myself work with the
horse from the ground or learn calmly together with others." — nonsense)
Q: Jak funguje Wi-Fi? ("How does Wi-Fi work?")
A: Wi-šinou je hlavně o koních, práci s koněm a bezpečném sedu.
(the word breaks apart mid-token: "Wi-mostly it's mainly about horses,
working with a horse and a safe seat.")
This is not a misconfiguration — it is expected behaviour for a 270M-parameter model trained on a narrow domain. The model cannot reliably say "I don't know"; it will attempt an answer in character whether or not it makes sense.
Limitations
- Not a source of facts. Answers about horses are usually sound, but they are not verified and the model can invent details. Do not use it for veterinary, medical or safety decisions.
- It falls apart out of domain. Anything outside horses, the stable and riding produces nonsense (see above).
- 270M parameters is small. It will not hold long context or multi-step reasoning, cannot
do arithmetic, and cannot summarise long texts. It was trained with
max_length=256. - Czech only. It was neither trained nor evaluated in any other language.
- A character, not an assistant. It answers in role, briefly and with attitude. It is not suited as a general-purpose chatbot.
- Quantisation hurts. The model is only ~540 MB in bf16 to begin with. Below Q8_0 quality degrades noticeably — small models tolerate quantisation far worse than large ones.
Out-of-scope use
This model is not intended to provide veterinary, medical, legal or safety advice, nor for use where an incorrect answer could lead to injury to a person or an animal. Working with horses carries real physical risk — consult a trainer or a veterinarian about specific situations, not a language model.
License
This model is derived from Gemma 3 and is governed by the Gemma Terms of Use, including the Prohibited Use Policy. By using the model you accept them.
Citation
@misc{svetlak2026pralinka,
title = {Gemma3-270M-Pralinka-V1.1},
author = {Světlák, Radim},
year = {2026},
url = {https://huggingface.co/RadimSvetlak/Gemma3-270M-Pralinka-V1.1}
}