GoLLeM v6 — 250M (Polish–English research base model)

STATUS: TRAINING COMPLETE — RESEARCH BASE MODEL. Final checkpoint of a single 760,000-step run. This is a base (pretrained) model, not instruction-tuned: it continues text and does not follow commands. An instruction-tuned version is a separate model with its own card. Results below use our protocol, not official benchmark submissions, and are compared only with our own earlier models (GoLLeM v4 and v5) measured the same way.

GoLLeM v6 is a 252.8M-parameter decoder-only language model trained from scratch on a Polish–English mix (about two thirds Polish, one third English). It is a Fabryka AI project (formerly SlayerLab).

Author: Arkadiusz Słota (Fabryka AI).

What it is for (and what it is not)

Intended use: research on small bilingual language models, comparison with GoLLeM v4 (Polish only) and v5 (English only), and text continuation in Polish and English.

Not intended for: production use, question answering without context, or following instructions. Like any base model of this size it has limited factual knowledge and may produce fluent but wrong text.

Results (final checkpoint, step 760,000)

All numbers: final checkpoint 221b1286…, single run, seed 1337. Our protocol, not official.

English — our implementation of the Glint leaderboard metrics (the same harness we used for GoLLeM v5):

eff = mean(ARC-Easy, BLiMP, WikiScore) × size multiplier, WikiScore = 100·(ln 500 − ln byte_ppl)/(ln 500 − ln 1.86), clipped to 0–100; the multiplier follows the leaderboard formula (1.0 at 150M parameters and above, 1.0084 for v5's declared 122.8M). Without the multiplier the English gap between v6 and v5 128M is −1.59 points, with it −2.23. v6 is below v5 128M in English: v6 saw about 8.9B English tokens (35.8 % of 24.9B), v5 128M was trained on English only.

Polish — our eff-PL protocol (definition fixed on 2026-09-29, before training started):

• PL clean removes MultiBLiMP pairs and ARC-Easy-PL questions found in GoLLeM v4's training corpus, so that v4 and v6 are compared on the same items: 2,207 of the 3,272 MultiBLiMP pairs are flagged by our scan of the v4 corpus, and pairs with a sentence shorter than 20 characters are also excluded (944 pairs remain); 14 ARC-Easy-PL test questions are in the v4 corpus (2,362 remain). "Clean" is therefore defined relative to the v4 corpus, not the v6 pool. The v6 pool was filtered before training against a protected list that includes these MultiBLiMP sentences (738 documents removed); a separate scan of the finished v6 pool against all 3,272 pairs has not been run. ± is one standard error (binomial, per axis, combined).
• Condition fixed in advance: PL clean ≥ v4 − 1 point. The condition and the two subsets were fixed on 2026-09-29, before any v6 checkpoint was evaluated in Polish (the subsets about one hour after training started); the threshold value, 68.62, depends only on the v4 measurement. The final checkpoint reaches 69.42, so the condition is met.
• Reading: in Polish, v6 is within one standard error of v4 (eff-PL −0.09, PL clean −0.20), while also being trained on about one third English. We do not claim it is better than v4 in Polish.
• W articles were selected by a fixed hash rule and removed if more than 5 % of their 13-grams appear in the v6 training pool or if they were on our privacy drop list (589 of 1,052 removed; 463 kept).

Training progress (evaluated every 6 data slots during training; intermediate checkpoints, not results):

The final checkpoint is the best of all 13 evaluated points on every axis; most of the late gain came during the learning-rate decay (last 20 % of steps). With a single run we cannot separate the effect of the decay from the effect of more training.

Training

Data queue with a rule. Training data arrived in 76 slots of 10,000 steps. After each slot the checkpoint was scored on five fixed control sets (English educational, encyclopedic and Q&A text; Polish encyclopedic and general text) on a separate machine. The first 6 slots ran in shadow mode: the rule was not applied, and its noise level was calibrated on them (per-set residual standard deviation σ, at least 0.002), so the rule's thresholds were set during the run, after the shadow slots. From slot 6 on, a slot would be rejected if the loss on any of the three English control sets rose by more than 3σ; the two Polish sets were scored and logged but did not decide. All 76 slots were accepted, none rejected.

Training data

GoLLeM v6 250M was pretrained on a fixed pool of 25,000,424,215 tokens (26,124,390 documents; tokenizer sha256 0640d3bd…); each token was seen at most once (24.90B of the 25.00B tokens were used). The pool is an aggregate of public sources; each source keeps its upstream terms.

• No non-commercial (NC) sources. Documents marked CC BY-NC-SA were removed from the English mix.
• Share-alike sources are about 7 % of tokens. The model weights are released under CC BY-SA 4.0.
• Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData (The Stack terms) and model-generated chat data (UltraChat, WildChat, tiny-textbooks: outputs of OpenAI models).
• Attribution: the English mix includes text from 55 OpenStax textbooks (CC BY 4.0, © Rice University, https://openstax.org), obtained from crumb/openstax-text (revision 8f502ca4…), modified (extracted, chunked, filtered). The full title list is in the appendix OpenStax attribution below, because the English mix itself is not published.
• Personal data: in Biblioteka Nauki, e-mail addresses were masked before training ([email]).
• Benchmark decontamination: documents matching WikiText-2 test/validation or ARC validation/test were removed from the English mix (4,362 documents). In the Polish part, documents containing an exact copy (after normalization) of any of 12,755 protected Polish evaluation sentences of at least 20 characters, including MultiBLiMP-pl, were dropped (738 documents).

Data availability

Usage

The model is a custom PyTorch architecture, not a transformers class. The repository contains:

# pip install torch safetensors tokenizers huggingface_hub
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("SlayerLab/GoLLeM-v6-250M")   # pin revision="<commit>" for reproducibility
sys.path.insert(0, path)
from modeling_gollem_v6 import load_gollem_v6, generate

model, tok = load_gollem_v6(path)                 # strict load, eval mode, CPU by default
print(generate(model, tok, "Ala ma kota", max_new_tokens=40, temperature=0.8, top_k=50, seed=1))
print(generate(model, tok, "Photosynthesis is the process by which", max_new_tokens=40))

• This is a base model: it continues text; it does not answer questions or follow instructions.
• Context: 1,024 tokens. RoPE uses the interleaved channel convention; if you port the model to another framework, keep that convention.
• The module definitions in modeling_gollem_v6.py are those used in training (initialisation and training loss removed). Loading model.safetensors into them reproduces the logits of the training checkpoint exactly (checked with torch.equal, also on the files downloaded from this repository).

Limitations

• Small base model: not instruction-tuned, limited factual knowledge, may reproduce biases and errors present in web data. Not for production use.
• Single run, single seed. Differences below about one standard error (or below the run-to-run noise we measured on smaller models, about 0.3 eff) should not be read as real.
• English and Polish scores use our protocol; they are not official leaderboard results, and self-reported numbers of other models measured differently are not comparable.
• Self-description. Prompted in chat format (ChatML), the model may describe itself as "an AI language model created by OpenAI". The English pretraining mix contains model-generated chat data (UltraChat, WildChat) with such self-descriptions. GoLLeM is not affiliated with OpenAI; it was built by Fabryka AI (author: Arkadiusz Słota).

License

Weights: CC BY-SA 4.0. Attribution: GoLLeM v6, Arkadiusz Słota / Fabryka AI, link to this repository; derivative weights under the same licence.

Training data keep their upstream licences; see Training data above.

Po polsku (skrót)

GoLLeM v6 250M to bazowy (niedostrojony do poleceń) model językowy polsko-angielski, 252,8 mln parametrów, wytrenowany od zera na 24,9 mld tokenów (ok. 2/3 polski, 1/3 angielski). Wyniki według naszego protokołu, nie oficjalne: eff (angielski) 74,71, eff-PL 79,77, PL czyste 69,42 ± 0,58. Warunek ustalony przed pierwszym pomiarem po polsku (PL czyste ≥ 68,62, czyli v4 − 1 pkt) jest spełniony. W polskim v6 jest w granicach jednego błędu standardowego od GoLLeM v4 (który trenowano tylko po polsku), w angielskim poniżej GoLLeM v5 128M (tylko angielski). Model do badań, nie do zastosowań produkcyjnych. W formacie czatu (ChatML) model może przedstawiać się jako „model językowy stworzony przez OpenAI”, bo angielska część danych zawiera rozmowy wygenerowane modelami OpenAI (UltraChat, WildChat). GoLLeM nie jest powiązany z OpenAI; stworzyła go Fabryka AI (autor: Arkadiusz Słota).

Appendix: OpenStax attribution

The training data of this model includes text extracted from the following OpenStax textbooks, each licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), © Rice University. Download for free at https://openstax.org. The texts were obtained from the Hugging Face dataset crumb/openstax-text (revision 8f502ca45f9f05cb5673eae445b7a97a4e8c4349). Modified: text extracted, chunked and filtered (chunks matching benchmark test/validation sets were removed).

Titles licensed CC BY-NC-SA 4.0, non-English titles, and three CC BY 4.0 titles whose text contains elements marked ‘CC BY-NC-SA’ (Introduction to Business, Organizational Behavior, Principles of Management) were not used.

GoLLeM v6 — Fabryka AI. Author: Arkadiusz Słota. Research base model (final checkpoint).