TokSuite – ByT5

Model Summary

TokSuite–ByT5 is part of TokSuite, a controlled suite of language models designed to isolate and measure the impact of tokenizer choice on language model behavior.

This model is architecturally identical to the other 13 TokSuite models and differs only in its tokenizer. All TokSuite models share:

• the same model architecture,
• the same training data,
• the same training budget,
• and a shared initialization for overlapping vocabulary items.

As a result, any behavioral differences observed between TokSuite models can be attributed directly to tokenizer design, rather than confounding factors such as data scale or optimization.

Tokenizer

• Tokenizer: ByT5
• Tokenization method: Byte-level
• Vocabulary size: 259
• Out-of-vocabulary handling: Byte-based (no OOVs)
• Language coverage: Language-agnostic
• Pretokenization source: None (raw bytes)

Processing details:

• Numbers: Represented as raw UTF-8 bytes
• Contractions: N/A
• Unicode normalization: None
• Whitespace / boundary markers: N/A
• Zerowidth chars: 3 Bytes

Why ByT5?

ByT5 represents an extreme point in the tokenizer design space: instead of learning subword units, it operates directly on bytes. This guarantees complete coverage of Unicode text and eliminates vocabulary fragmentation issues common in subword tokenizers.

Model Architecture

• Architecture: Decoder-only Transformer (Lingua's Llama-3.2-1B configuration)
• Non-embedding parameters: ~1B
• Context length: 4096 tokens
• Framework: Meta Lingua
• Initialization: Shared super-vocabulary initialization across TokSuite models

The architecture and training setup are identical across all TokSuite models; only the tokenizer differs.

Training Data

The model was trained on a multilingual corpus totaling approximately 100B tokens, consisting of:

• English: 40B tokens from FineWeb-Edu
• Multilingual: 60B tokens evenly distributed across:
  • Chinese (ZH)
  • Turkish (TR)
  • Italian (IT)
  • Farsi (FA)

You can find the pretraining dataset here: toksuite/toksuite_pretraining_data

Because all models are trained with a fixed token budget, the byte-level tokenizer processes significantly more raw text (in bytes) than subword-based tokenizers. This reflects a realistic trade-off between compression efficiency and robustness.

Training Procedure

• Training steps: 100,000
• Sequence length: 4096
• Batch size: 256 sequences
• Optimizer: AdamW
• Peak learning rate: 1e-3
• Schedule: Cosine decay with 2,000 warm-up steps
• Weight decay: 0.1

Evaluation

Canonical Benchmarks

The model was evaluated on standard base-language-model benchmarks:

• HellaSwag
• ARC
• PIQA
• XNLI

These evaluations verify that the model exhibits reasonable base language modeling behavior at its scale and training budget.

TokSuite Robustness Benchmark

TokSuite–ByT5 is evaluated on the TokSuite multilingual robustness benchmark, which probes real-world perturbations including:

• orthographic and spelling errors,
• diacritics presence/absence,
• keyboard and input-method noise,
• Unicode styling and homoglyphs,
• OCR and spacing artifacts,
• LaTeX and STEM formatting.

Tokenization Robustness under Multilingual Text Perturbations
Values represent relative performance drop, computed as (Acc_clean − Acc_perturbed) / Acc_clean, where lower values indicate greater robustness.

Perturbation types include:

• Input: non-native keyboard input and romanization
• Diacr.: optional diacritics
• Orth.& Gram.: orthographic and grammatical errors
• Morph: morphological variations including derivations, inflections, and contractions
• Noise: homoglyph substitutions, OCR artifacts, typos, and spacing errors
• LaTeX: LaTeX-style mathematical formatting
• STEM: scientific diagrams and notational conventions
• Unic.: Unicode styling characters

NEN denotes non-English inputs and EN denotes English inputs. The Avg column reports the average relative performance drop across all perturbation categories.

Tokenizer-Specific Findings

In TokSuite evaluations, the ByT5 tokenizer exhibits:

• Strong robustness to:
  • multilingual noise,
  • OCR errors,
  • keyboard perturbations,
  • Unicode formatting and homoglyphs.
• Particularly strong performance in Turkish and Chinese settings, including romanized inputs.
• Minimal catastrophic failure modes, as byte corruption results in predictable token sequences rather than invalid subwords.

However, this robustness comes at a cost:

• higher sequence lengths,
• reduced tokenization efficiency,
• increased computational overhead per example.

These findings highlight a fundamental robustness–efficiency trade-off in tokenizer design.

Intended Use

This model is intended for:

• research on tokenization and robustness,
• multilingual NLP analysis,
• controlled ablation studies,
• benchmarking tokenizer behavior under noise.

It is not instruction-tuned, aligned, or optimized for deployment.

Limitations

• Trained on a limited set of five languages.
• Not optimized for instruction following or dialogue.
• Fixed token budget constrains exposure to raw text depending on tokenization efficiency.
• Intended strictly for research purposes.

Ethical Considerations

TokSuite models are released strictly for research purposes.
They inherit biases present in large-scale web data and should not be used in high-stakes or sensitive applications without additional alignment and evaluation.

Citation

If you use this model, please cite:

@article{toksuite2025,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altıntaş, Gul Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  year={2025},
  arxiv={https://arxiv.org/abs/2512.20757},
}