wieszcz-xix-349m

A 349M-parameter decoder-only language model trained from scratch on a corpus of Polish text published between 1800 and 1918. It writes in the language of that period, in pre-reform orthography. It is a base model: it continues text and does not answer questions.

This is the 349M rung of a three-model ladder (47M, 107M, 349M) trained on one frozen corpus with the same steps, batch and seed, so the rungs differ only in parameter count. The smaller rungs write the same period Polish but lose track of what they are talking about within a few sentences, so this repository carries the rung worth using; the whole ladder, including the pre-decay and epoch-one checkpoints the paper's epochs table reads, is archived at doi:10.5281/zenodo.22099358. The corpus is SKocur/polish-pre1918-corpus.

⚠️ Read Intended use and content warning before loading this model. It reproduces the prejudices of nineteenth-century print.

What it scores

Dense evaluation on the held-out split of its own corpus, 8,192 windows of 1,024 tokens in fp32, covering the whole split:

Scored under the same protocol, the smaller rungs reach 2.905 (107M) and 3.101 (47M) nats, which is what the paper's ladder measures.

Architecture

Llama and Gemma-2 lineage: RoPE, RMSNorm applied before and after each sublayer, QK-Norm, SwiGLU, grouped-query attention, weight tying. Trained with Muon on the 2D hidden weights and AdamW on embeddings and norm gains.

The tokenizer is part of the model. It was trained on nineteenth-century Polish, and a modern Polish tokenizer shreds pre-reform spelling, so it ships here in both forms: tokenizer.json for AutoTokenizer, and the vocab.json and merges.txt it was built from. All three are hashed in SHA256SUMS.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "SKocur/wieszcz-xix-349m"
tok = AutoTokenizer.from_pretrained(name)
net = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)

ids = tok("Doniesiono nam z Warszawy, iż", return_tensors="pt")
out = net.generate(**ids, max_new_tokens=200, do_sample=True, temperature=0.9, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code is needed because this architecture is not one the library ships. The code it runs is modeling_wieszcz.py in this repository, a short wrapper that adds the library's interface around model.py and no arithmetic of its own. Both files are meant to be read before they are run.

Sampling through the wrapper recomputes the whole prefix at every step, because the KV cache in model.py predates the library's cache protocol and does not implement it. For long or repeated generation, load the network directly and use its own cached sampler:

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_model
from tokenizers import ByteLevelBPETokenizer

path = snapshot_download("SKocur/wieszcz-xix-349m")
sys.path.insert(0, path)
from model import GPT

net = GPT(json.load(open(f"{path}/config.json")))
load_model(net, f"{path}/model.safetensors")
net.eval()

tok = ByteLevelBPETokenizer(f"{path}/vocab.json", f"{path}/merges.txt")
ids = torch.tensor([tok.encode("Doniesiono nam z Warszawy, iż").ids])
out = net.generate(ids, max_new_tokens=200, temperature=0.9, top_p=0.9)
print(tok.decode(out[0].tolist()))

That path needs only torch, safetensors and tokenizers, and it reads the same model.safetensors through the file that trained the weights. The build checks that the two paths return identical logits.

Intended use and content warning

Research and digital humanities: period language, register and discourse, historical NLP, scaling and data-constraint experiments.

This model reproduces antisemitic, nationalist, colonial and misogynist discourse that was ordinary in nineteenth-century Polish print. The prejudice comes from the sources, and the training preserved it, because a model of a period that had been cleaned of its prejudices would model a century that never existed. The accompanying paper documents unprompted prejudiced output from these checkpoints and reports a first prevalence estimate.

Do not put this model in front of end users, interactively or through generated text presented as information, without a filtering layer, visible framing that identifies the output as a historical-language simulation, and your own legal review. It is not an authority on any subject. Full conditions are in TERMS-OF-USE.md.

Limitations

The context is 1024 tokens, which bounds both coherence and any interactive use. The corpus is OCR of period scans, so the model learned some of that damage and occasionally writes it back. Generation stays in period register by construction, which means a modern prompt does not produce a modern answer.

The temporal bound has a measured residue. 90 documents of the training split, all from the transcribed Wolne Lektury source and 0.04% of the corpus bytes, are later than 1918: 79 modern translations and eleven volumes of Proust in a translation of the 1930s. Up to 0.38% of the bytes are translations or works by people who lived past 1918. The paper's audit section gives the counts.

Citation

@misc{kocur2026wieszcz,
  title  = {Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch},
  author = {Kocur, Szymon},
  year   = {2026},
  eprint = {2610.10592},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/abs/2610.10592},
}

Paper: arXiv:2610.10592.

Corpus: doi:10.5281/zenodo.22099302. Weights archive with all nine checkpoints: doi:10.5281/zenodo.22099358.