futolm Czech and Slovak models
Separate 29.37M-parameter Czech and Slovak language models for next-word prediction in FUTO Keyboard. All files are F16 FUTO KeyboardLM GGUF exports with the shared SentencePiece tokenizer embedded.
For typing, use the synth models: cs_synth.gguf (Czech) or sk_synth.gguf (Slovak). They continue the chat models on synthetic text messages and predict everyday messages best. The chat and base models are kept for comparison and further training.
Synth models
| Model | Download | FUTO language metadata | Initialization | Training tokens in this run | Final step |
|---|---|---|---|---|---|
| Czech synth | cs_synth.gguf | cs | Czech chat | 30,932,992 | 236 |
| Slovak synth | sk_synth.gguf | sk | Slovak chat | 32,505,856 | 248 |
About one million words of everyday text messages per language were generated with OpenAI gpt-6-luna (Codex CLI), with varied recipients, topics, registers and lengths: 108,431 unique Czech and 111,329 unique Slovak messages. Each generation call is one split unit, so held-out messages never come from a call that produced training text. The training mix is 25% synthetic messages, with the chat mix (subtitles, forum pages, FineWeb2) for the rest, about four passes over the synthetic training split. Peak LR 5e-5, 50 warmup steps, cosine decay to 10%.
| Language | Data | Chat top-1 / top-3 | Synth top-1 / top-3 |
|---|---|---|---|
| Czech | held-out synthetic messages | 13.8 / 22.8 | 20.6 / 32.6 |
| Czech | chat probes | 41.8 / 64.1 | 50.9 / 71.8 |
| Czech | subtitles | 23.0 / 33.6 | 22.0 / 32.4 |
| Czech | FineWeb2 | 21.4 / 33.6 | 21.4 / 33.4 |
| Slovak | held-out synthetic messages | 14.8 / 24.2 | 24.8 / 36.0 |
| Slovak | chat probes | 42.3 / 64.5 | 53.6 / 70.5 |
| Slovak | subtitles | 24.0 / 35.2 | 24.2 / 34.6 |
| Slovak | FineWeb2 | 20.2 / 31.8 | 20.6 / 32.0 |
Subtitle and FineWeb2 differences are within noise (about ±4 points with 500 samples). The probes and the synthetic messages were both written by OpenAI models, and most probe phrases (for example "Dobrou noc") also occur in the synthetic text, so the probe gain partly reflects shared phrasing; the held-out synthetic messages, subtitles, and FineWeb2 are the less biased comparisons.
Chat models
| Model | Download | FUTO language metadata | Initialization | Training tokens in this run | Final step |
|---|---|---|---|---|---|
| Czech chat | cs_chat.gguf | cs | Czech base | 1,000,341,504 | 7,632 |
| Slovak chat | sk_chat.gguf | sk | Slovak base | 1,000,341,504 | 7,632 |
The mix is OpenSubtitles v2024 dialogue, discussion-forum pages selected from FineWeb2, and FineWeb2 replay: Czech 50/15/35, Slovak 48/12/40. Peak LR 1.5e-4, 300 warmup steps, cosine decay to 10%.
Whole-word next-word prediction, model only (FUTO's dictionary not involved). Probes are 220 handwritten chat prefixes per language; subtitles and FineWeb2 are 500 held-out word positions (about ±4 points).
| Language | Data | Base top-1 / top-3 | Chat top-1 / top-3 |
|---|---|---|---|
| Czech | chat probes | 22.7 / 43.6 | 41.8 / 64.1 |
| Czech | subtitles | 16.6 / 24.0 | 23.0 / 33.6 |
| Czech | FineWeb2 | 22.0 / 34.6 | 21.4 / 33.6 |
| Slovak | chat probes | 24.6 / 38.6 | 42.3 / 64.5 |
| Slovak | subtitles | 16.2 / 28.0 | 24.0 / 35.2 |
| Slovak | FineWeb2 | 21.0 / 35.4 | 20.2 / 31.8 |
Examples (top three): Ahoj, jak se → máš, máte, vede; Dobrou → noc, chuť, zprávou; Ahoj, ako sa → máš, máte, darí.
The probes were written by an AI model and have not been reviewed by a native speaker. OpenSubtitles has no blanket licence; these weights are shared for personal and research use.
Base models
| Model | Download | FUTO language metadata | Initialization | Training tokens in this run | Final step | Latest logged validation loss |
|---|---|---|---|---|---|---|
| Czech base | cs_base.gguf | cs | From scratch | 3,000,238,080 | 22,890 | 2.7750 at step 22,499 |
| Slovak base | sk_base.gguf | sk | Czech base weights | 3,001,024,512 additional Slovak tokens | 22,896 | 2.6691 at step 22,499 |
Download the file for your language and import it through FUTO Keyboard's model import flow. The models are separately selectable; each declares its own language.
Model and training
| Property | Value |
|---|---|
| Architecture | Llama, 8 layers, hidden size 512, FFN size 1,024, 8 attention/KV heads |
| Parameters per model | 29,368,832 |
| Vocabulary | 8,192-piece SentencePiece Unigram, shared Czech/Slovak tokenizer |
| Context capability | 2,048 positions; trained on 512-token sequences |
| Czech data | Cleaned Czech FineWeb2 (ces_Latn) |
| Slovak data | Cleaned Slovak FineWeb2 (slk_Latn) |
| FineWeb2 revision | af9c13333eb981300149d5ca60a8e9d659b276b9 |
| Export | GGUF v3, F16 matrices and F32 normalization weights |
| Size of each GGUF | 59,313,504 bytes (59.3 MB) |
| FUTO features | base_v1 inverted_space |
Slovak continued pretraining starts from the clean Czech base with the same tokenizer and architecture. It uses a fresh optimizer, schedule, and Slovak data position. The Slovak token count above excludes the Czech pretraining tokens inherited through initialization. Its peak learning rate was 1e-4, with 300 warmup steps and cosine decay to 1e-5.
Verification
All six files passed structural checks and loaded with FUTO Keyboard's native code pinned to commit 0246cbe2fb2f130a5dc475c508215760044dea20 (version 0.1.30). Each model's native inference was compared with its PyTorch export across 11 cases: eight next-word prompts and three embedding-mix cases. All top-1 predictions matched and cosine similarity was at least 0.999. Top-five token sets matched in every case except one embedding-mix case of cs_synth (4 of 5, a near-tie at fifth place); embedding mixing is not used by these models' declared features.
See the verification reports (cs_base, sk_base, cs_chat, sk_chat, cs_synth, sk_synth) and SHA256SUMS.
Limitations
Keyboard-task autocorrect fine-tuning has not been performed, and autocorrect features are not declared. Embedding-mix verification checks numerical compatibility, not correction quality. Phone import, latency, and typing quality have not yet been tested.