Bielik-4.5B-v3 · matura-history SFT v2 ("Mały, ale wariat")

Full-parameter SFT of speakleash/Bielik-4.5B-v3.0-Instruct on synthetic tasks and model essays in the format of the Polish extended-level history matura (CKE). Built for the "Mały, ale wariat" track of the SlayerLab hackathon: the smallest model file that scores at least 35% on the team's history-matura evaluator. Same recipe and the same 4,144 training rows as the 1.5B sibling, SlayerLab/bielik-1.5b-v3-matura-history-sft2.

Registered candidate: sft45-IQ3_XXS.gguf, 1,851,896,576 bytes (1.85 GB). It is the smallest file that passes both the team evaluator (harness v5) and an unseen CKE May 2026 paper (with and without the harness). The untouched base at the same 3-bit size fails the past-exam track (18/55); the SFT is what brings it to 22/55.

This repo holds the bf16 safetensors (Transformers format, 2 shards), the llama.cpp GGUF quants made with a Polish importance matrix, the imatrix itself, and the training logs. Code, data recipe and every evaluation run: slayerlabs/hackathon → maly-ale-wariat/ (MODEL_CARD.md, RESULTS.md, REPORT.md).

Files

Quants were made with llama.cpp build 11146 (commit 7fe450e19) via train/quantize.sh, with the same Polish calibration text as the 1.5B (calib-v1.txt).

How to run

The configuration that was scored (llama.cpp llama-server, OpenAI-compatible API on port 8891):

hf download SlayerLab/bielik-4.5b-v3-matura-history-sft2 sft45-IQ3_XXS.gguf --local-dir .
llama-server -m sft45-IQ3_XXS.gguf --port 8891 --ctx-size 8192 -ngl 99 --flash-attn on --jinja \
  --chat-template-kwargs '{"enable_thinking":false}'

llama-server -hf SlayerLab/bielik-4.5b-v3-matura-history-sft2:IQ3_XXS and ollama run hf.co/SlayerLab/bielik-4.5b-v3-matura-history-sft2:IQ3_XXS also work.

The reported harness scores used matura-harness in front of that server with the flags in maly-ale-wariat/eval/harness_args_v5.txt (evaluator-style prompts for source-based tasks, plwiki BM25 retrieval of 2 × 700 characters only for tasks of 600 characters or less and for essays, strict closed-answer normalization, repetition trimming, DRY sampling and a 380-word floor for essays). The 1.7 GB retrieval index is not part of the model; the steps to rebuild it are in the repo README.

Transformers (bf16 safetensors):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "SlayerLab/bielik-4.5b-v3-matura-history-sft2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "Jesteś zdającym maturę rozszerzoną z historii. Odpowiadasz po polsku, zwięźle i konkretnie. "
                                  "Najważniejsze są źródła podane w zadaniu; materiały pomocnicze mogą być nietrafne. "
                                  "Wykonaj dokładnie polecenie i podaj wymaganą liczbę elementów."},
    {"role": "user", "content": "Podaj nazwę bitwy z 1410 r., w której wojska polsko-litewskie pokonały zakon krzyżacki, "
                                "i wymień jednego z dowódców strony polsko-litewskiej."},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

The system prompt above is the one used in training. Chat template: Bielik ChatML (<s><|im_start|>role\n…<|im_end|>), no thinking mode. Context: 32768 tokens (the scored runs used 8192).

Evaluation

Team evaluator history-matura-eval: two tracks of 55 points each, 34 items adapted from the CKE May 2023 extended-level history paper ("past exam") and 34 Wikipedia-based items, graded by a deterministic closed-answer scorer plus an LLM judge (google/gemini-3.1-flash-lite). Pass = at least 20/55 (35%) on each track. Two protocols:

• raw: the evaluator's bare-model runner (its system prompt, the task text only, the answer submitted verbatim);
• harness v5: matura-harness with eval/harness_args_v5.txt, as described above.

A third, truly unseen paper, CKE May 2026 (39 items, 60 points, pass = 21), was converted to the evaluator's item format and scored with the same scorer and judge. Scores below are strict re-scores (eval/strict_rescore.py); the value the evaluator reported is in brackets where it differs.

¹ The evaluator reports no total when the judge errs or grades an item "review". ² mradermacher's imatrix quant of the ungated mirror, used as the size-matched untouched baseline.

What this shows: the untouched 4.5B has real margin at Q8 (5 GB) but loses most of it below 4 bits (18/55 at IQ3_XXS). The SFT recovers the past-exam track at that size (22/55) and holds 22–23/60 on the unseen 2026 paper, where every 1.5B variant sits at 19–21. The price is a file twice the size of the 1.5B candidate.

Caveats

• A single evaluator run varies by about ±3 points; the judge is an LLM. The 22/55 past-exam pass has a 2-point margin.
• The bare-runner past-exam score for IQ3_XXS is unavailable (judge error voided the track), so "passes under both protocols" rests on the Wikipedia track and the 2026 paper for the bare protocol.
• IQ2_M fails; Q3_K_M was not evaluated.
• Known failure modes: true/false statements near chance level, open answers naming the wrong person/place/date, essays with factual errors.

Training

Data provenance and licensing

• Weights: Apache-2.0, same as the base model.
• The synthetic tasks and essays were written by Claude (Anthropic) via Claude Code workflows following data/gen/STYLE.md, fact-checked by a second agent pass and validated by data/gen/validate_items.py. They are CC BY-SA 4.0-compatible.
• Retrieved evidence comes from Polish Wikipedia (CirrusSearch dump 2026-09-20, CC BY-SA 4.0); every record keeps its source URL.
• The CKE 2024/2025 items are not redistributed (CKE copyright). The evaluator's questions and keys were never used for training; build_sft.py drops any record sharing word 8-grams with them, plus essays on the held-out themes.

Intended use and limitations

A hackathon research artifact: a small Polish model that answers exam-style history questions in the CKE matura format. It hallucinates names, dates and places, answers true/false items near chance, and was tuned for a specific evaluator. Do not rely on it for factual information or grading.

Acknowledgements

SpeakLeash for Bielik-4.5B-v3.0-Instruct; llama.cpp for quantization and inference; Polish Wikipedia authors (CC BY-SA 4.0).