Functionalizer 100M model
This is a 100M example model. Each model is trained on the dataset specified in the name for 50000 epochs with the given seed.
This model is presented in the paper The Functionalizer: Lossless Functional Decomposition for Subword Tokenization.
The model was trained using the Functionalizer framework. Training code and detailed performance analysis are available: https://github.com/connor-makowski/functionalizer
Running the model
To run the model, use the custom tokenizer fork available here: https://github.com/connor-makowski/tokenizers/tree/functionalizer
Citation
If you find this model or the Functionalizer framework useful, please cite:
@misc{makowski2026functionalizerlosslessfunctionaldecomposition,
title={The Functionalizer: Lossless Functional Decomposition for Subword Tokenization},
author={Connor Makowski and Willem Guter},
year={2026},
eprint={2609.15991},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.15991},
}