BDH-25M-PL — Polish Looped-Transformer (Fast-Weight) Language Model

A small, open-weight language model built on the BDH (Fast Weight Layers) architecture, trained on a clean Polish mix from SlayerLab/polish-dynaword-mix (100M tokens, byte-level).

Abstract (EN)

BDH-25M-PL is a small, open-weight language model built on the BDH (Fast Weight Layers) architecture, drawn from the family of looped models with localized latent recurrence. Unlike a standard transformer, BDH shares its representation as both key and value (Q==K), so the internal state acts as a working memory that adapts to context on the fly, without changing weights. The model operates directly on UTF-8 bytes (vocab 256, no tokenizer) and was trained on a clean, diversified Polish corpus, SlayerLab/polish-dynaword-mix (100M tokens, legal content capped at ~7%). It is a baseline: it confirms the BDH architecture trains and generates correctly end-to-end, producing grammatically correct Polish sentences. Per scaling laws, a 25M-parameter model is undertrained for high quality (it would need ~0.5–2B tokens), yet it serves as a verifiable, public starting point for larger variants.

Abstract (PL)

BDH-25M-PL to mały, otwarty model językowy oparty na architekturze BDH (Fast Weight Layers, ang. warstwy szybkich wag), który wprowadzono w rodzinie modeli pętlowanych z lokalną rekurencją ukrytą (looped latent recurrence). W przeciwieństwie do klasycznego transformera, BDH współdzieli reprezentację jako klucz i wartość (Q==K), dzięki czemu wewnętrzny stan staje się swego rodzaju pamięcią roboczą adaptującą się do kontekstu w locie, bez zmiany wag. Model działa bezpośrednio na bajtach UTF-8 (vocab 256, brak tokenizera) i został wytrenowany na czystym, zróżnicowanym polskim zbiorze SlayerLab/polish-dynaword-mix (100 mln tokenów, treść prawna ograniczona do ~7%). Jest to baseline: potwierdza, że architektura BDH trenuje i generuje poprawnie end-to-end, generując gramatycznie poprawne polskie zdania. Zgodnie z prawami skalowania model o rozmiarze 25M param. jest niedoćwiczony dla wysokiej jakości (wymagałby ~0.5–2 mld tokenów), pełni jednak rolę weryfikowalnego, publicznego punktu startowego pod większe warianty.

Training curves

Train and validation loss over 10,000 byte-level steps (final val loss ≈ 1.41; random-init baseline ≈ 5.6).

Architecture

• BDH from pathwaycom/bdh — looped latent recurrence / fast weights
• n_layer=8, n_embd=256, n_head=4, mlp_internal_dim_multiplier=128
• Vocab: byte-level (vocab 256) — no tokenizer, works directly on UTF-8 bytes
• ~25.3M parameters · seq length 2048
• Key feature: Q==K (fast weights) — the shared state acts as both key and value (localized latent recurrence)

Training

• Data: SlayerLab/polish-dynaword-mix (100M tokens, law cap 7%, dedup, cleaned)
• Steps: 10000 · final val loss ≈ 1.41 · byte-level (random-init baseline ≈ 5.6)
• Optimizer: ZClip + protocol B.2 (as in the BDH paper)

Note on scale

This is a 25M model trained on 100M tokens — per scaling laws it is undertrained for high quality, but it serves as a working baseline: it generates grammatically-correct Polish and demonstrates that the BDH architecture trains and generates correctly end-to-end.

Quick start

import torch
from safetensors.torch import load_file
from bdh import BDH, BDHConfig

cfg = BDHConfig(n_layer=8, n_embd=256, n_head=4,
                mlp_internal_dim_multiplier=128, dropout=0.1, vocab_size=256)
model = BDH(cfg)
model.load_state_dict({k.replace("model.", ""): v
                       for k, v in load_file("model.safetensors").items()}, strict=True)
model.eval()

# byte-level prompt (no tokenizer)
prompt = "Warszawa jest stolicą Polski i "
ids = torch.tensor([list(prompt.encode("utf-8"))])
# ... autoregressive loop, one byte at a time with softmax(temp)

Files

• model.safetensors — weights (101MB)
• config.json — hyperparameters and training metadata

License

Weights: CC-BY-4.0. Architecture: MIT (pathwaycom/bdh).