TokenizerTest-32m-B (Polish research base model, tokenizer comparison — arm B)

A small Polish base language model trained as arm B of a controlled tokenizer experiment. It is a research checkpoint for comparing tokenizers, not a model for use. It is a Fabryka AI project (formerly SlayerLab).

Author: Arkadiusz Słota (Fabryka AI), who designed and ran the experiment. Tokenizer of this arm: stubbornGuy.

Companion model (tokenizer A/B experiment)

This model is one half of a controlled tokenizer comparison. Two models were trained with the same architecture, training recipe, seed and the same Polish text. The only difference is the tokenizer:

This model: B.

Shared recipe: ~38.9M parameters (tied input/output embeddings); "32m" in the repo name is the recipe name ("Glint 32M r6"), not the parameter count. Architecture: 15 layers, d_model 384, 6 heads, context 1024, Muon optimizer, batch 32, seed 1337, trainer train_gpt_ref_r6.py (0223c083).

Both arms saw the same ~12.4 GB of Polish text (public dataset SlayerLab/slayer-pl-8x3b @ d50df04a, files pack-00, pack-01 and shared/{core,core_sa,legal,legal_sa}; validation = pack-07 after 13-gram decontamination); B splits it more finely (on the training text 3.96 vs 4.25 bytes per token incl. EOS; on validation 4.14 vs 4.41), so it needed more steps for the same bytes. For the same reason B has lower bits per token (val 4.395 vs 4.668), which says nothing about model quality.

How to compare: loss per token is not comparable across tokenizers. We compare bits per byte on the same bytes (lower is better):

From ~4 B bytes onward the gap is roughly constant (val +0.003…+0.004, W +0.008…+0.010) and does not close by the end of training. Curve every 10k steps (bits per byte; note: at the same step B has seen fewer bytes than A):

The same numbers, with checkpoint and data hashes, are in bpb_curve.json.

Caveat: one seed per arm, no noise estimate. The honest reading is "at this scale the B tokenizer brings no gain in bits per byte", not "B is worse".

Training data

Public dataset SlayerLab/slayer-pl-8x3b at revision d50df04a; the same files for both arms:

Usage

The model is a custom PyTorch architecture (same code as SlayerLab/GoLLeM-v6-250M), not a transformers class.

# pip install torch safetensors tokenizers huggingface_hub
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("SlayerLab/TokenizerTest-32m-B")   # pin revision="<commit>" for reproducibility
sys.path.insert(0, path)
from modeling_gollem_v6 import load_gollem_v6, generate

model, tok = load_gollem_v6(path)                          # final checkpoint
print(generate(model, tok, "Kraków jest miastem", max_new_tokens=40, temperature=0.8, top_k=50, seed=1))
model_10k, _ = load_gollem_v6(f"{path}/step_10000")        # an intermediate checkpoint

Loading any model.safetensors of this repository into modeling_gollem_v6.py reproduces the logits of the corresponding training checkpoint exactly (checked with torch.equal for every step).

Limitations

• Small base model (~38.9M parameters): it continues Polish text; it does not answer questions or follow instructions. Limited knowledge; may reproduce errors and biases of web text. Research use only.
• Single run, single seed per arm; see the caveat above.

License

Weights: CC BY-SA 4.0 (the training text includes CC BY-SA parts of SlayerLab/slayer-pl-8x3b). Attribution: TokenizerTest-32m-B, Arkadiusz Słota / Fabryka AI, link to this repository; derivative weights under the same licence. The tokenizer (tokenizer.json) was created by stubbornGuy and is released here under CC BY-SA 4.0 with his consent. Training data keep their upstream licences; see the dataset card of SlayerLab/slayer-pl-8x3b.

TokenizerTest — Fabryka AI. Author: Arkadiusz Słota. Tokenizer of arm B: stubbornGuy.