Bielik-1.5B-v3 · matura-history SFT v2 ("Mały, ale wariat")

Full-parameter SFT of speakleash/Bielik-1.5B-v3.0-Instruct on synthetic tasks and model essays in the format of the Polish extended-level history matura (CKE). Built for the "Mały, ale wariat" track of the SlayerLab hackathon: the smallest model file that scores at least 35% on the team's history-matura evaluator.

Registered candidate: sft2-IQ4_XS.gguf, 868,041,888 bytes (0.87 GB). It is the smallest file that passes the team evaluator, when run through the matura-harness v5 retrieval setup. Read the caveats below before quoting the number. The 4.5B sibling that also passes an unseen 2026 paper is SlayerLab/bielik-4.5b-v3-matura-history-sft2.

This repo holds the bf16 safetensors (Transformers format), the llama.cpp GGUF quants made with a Polish importance matrix, the imatrix itself, and the training logs. Code, data recipe and every evaluation run: slayerlabs/hackathon → maly-ale-wariat/ (MODEL_CARD.md, RESULTS.md, REPORT.md).

Files

PPL = perplexity on held-out Polish text (calib/ppl-heldout.txt, 40 × 1024 tokens); lower is better. IQ4_XS with the Polish imatrix is effectively lossless against Q8_0; every 3-bit variant failed the evaluator. Quants were made with llama.cpp build 11146 (commit 7fe450e19) via train/quantize.sh.

How to run

The configuration that was scored (llama.cpp llama-server, OpenAI-compatible API on port 8891):

hf download SlayerLab/bielik-1.5b-v3-matura-history-sft2 sft2-IQ4_XS.gguf --local-dir .
llama-server -m sft2-IQ4_XS.gguf --port 8891 --ctx-size 8192 -ngl 99 --flash-attn on --jinja \
  --chat-template-kwargs '{"enable_thinking":false}'

llama-server -hf SlayerLab/bielik-1.5b-v3-matura-history-sft2:IQ4_XS and ollama run hf.co/SlayerLab/bielik-1.5b-v3-matura-history-sft2:IQ4_XS also work.

The reported scores used matura-harness in front of that server with the flags in maly-ale-wariat/eval/harness_args_v5.txt (evaluator-style prompts for source-based tasks, plwiki BM25 retrieval of 2 × 700 characters only for tasks of 600 characters or less and for essays, strict closed-answer normalization, repetition trimming, DRY sampling and a 380-word floor for essays). The 1.7 GB retrieval index is not part of the model; the steps to rebuild it are in the repo README.

Transformers (bf16 safetensors):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "SlayerLab/bielik-1.5b-v3-matura-history-sft2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "Jesteś zdającym maturę rozszerzoną z historii. Odpowiadasz po polsku, zwięźle i konkretnie. "
                                  "Najważniejsze są źródła podane w zadaniu; materiały pomocnicze mogą być nietrafne. "
                                  "Wykonaj dokładnie polecenie i podaj wymaganą liczbę elementów."},
    {"role": "user", "content": "Podaj nazwę bitwy z 1410 r., w której wojska polsko-litewskie pokonały zakon krzyżacki, "
                                "i wymień jednego z dowódców strony polsko-litewskiej."},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

The system prompt above is the one used in training. Chat template: Bielik ChatML (<s><|im_start|>role\n…<|im_end|>), no thinking mode. Context: 8192 tokens.

Evaluation

Team evaluator history-matura-eval: two tracks of 55 points each, 34 items adapted from the CKE May 2023 extended-level history paper ("past exam") and 34 Wikipedia-based items, graded by a deterministic closed-answer scorer plus an LLM judge (google/gemini-3.1-flash-lite). Pass = at least 20/55 (35%) on each track. Two protocols:

• raw: the evaluator's bare-model runner (its system prompt, the task text only, the answer submitted verbatim);
• harness v5: matura-harness with eval/harness_args_v5.txt, as described above.

A third, truly unseen paper, CKE May 2026 (39 items, 60 points, pass = 21), was converted to the evaluator's item format and scored with the same scorer and judge. Scores below are strict re-scores (eval/strict_rescore.py); the value the evaluator reported is in brackets where it differs.

¹ The evaluator reports no total when the judge errs or grades an item "review".

What the SFT changed: bare Bielik-1.5B scores 15–17/55 on the past exam, just under the threshold. SFT on about 4k synthetic and CKE-style tasks lifts past-exam open answers from 7–8 to 14–16 points, which is what makes the model pass.

Caveats

• Judge glitch (+2) in the headline run. On one 3-statement true/false item the model answered only "P"; the judge wrote "incomplete" but filled the full correct key, and the scorer counted 2 points. The corrected past-exam score is 23/55 (41.8%), still a pass with a 3-point margin.
• The 0.87 GB file passes only through the harness. Under the evaluator's bare runner it scores 18/55 on the past exam. The harness adds points through retrieval on short tasks, essay handling and strict closed-answer formatting. A dashboard that runs the bare GGUF will show this file failing; the 0.97 GB Q4_K_M passes under both protocols.
• A single evaluator run varies by about ±3 points; the judge is an LLM and a small model's answer can flip between near-identical prompts.
• On the unseen CKE May 2026 paper every 1.5B variant, the untouched base included, lands at 19–21/60, right at the 35% line. The fine-tuning gain shows on the evaluator's paper, not on this one.
• Known failure modes: true/false statements at chance level, open answers naming the wrong person/place/date, essays that hallucinate facts.

Training

Things that did not help (see RESULTS.md): continued pretraining on 39M plwiki tokens, more real CKE papers, 4 epochs at lr 2e-5, a uniform model soup, reason-first prompting or self-consistency voting for closed questions.

Data provenance and licensing

• Weights: Apache-2.0, same as the base model.
• The synthetic tasks and essays were written by Claude (Anthropic) via Claude Code workflows following data/gen/STYLE.md, fact-checked by a second agent pass and validated by data/gen/validate_items.py. They are CC BY-SA 4.0-compatible.
• Retrieved evidence comes from Polish Wikipedia (CirrusSearch dump 2026-09-20, CC BY-SA 4.0); every record keeps its source URL.
• The CKE 2024/2025 items are not redistributed (CKE copyright). The evaluator's questions and keys were never used for training; build_sft.py drops any record sharing word 8-grams with them, plus essays on the held-out themes.

Intended use and limitations

A hackathon research artifact: a small Polish model that answers exam-style history questions in the CKE matura format. It hallucinates names, dates and places, answers true/false items near chance, and was tuned for a specific evaluator. Do not rely on it for factual information or grading.

Acknowledgements

SpeakLeash for Bielik-1.5B-v3.0-Instruct; llama.cpp for quantization and inference; Polish Wikipedia authors (CC BY-SA 4.0).