Clio (NovelAI Legacy Model)

Clio is the first fully original large language model that we pretrained from scratch on our Shoggy H100 cluster. At 3 billion paramters, she runs very fast, but thanks to the, for her time, very large training volume, her writing outperforms much bigger models like Euterpe and Krake. She was trained on our custom Nerdstash dataset, with our custom Nerdstash Tokenizer V1.

Back when she was released, Clio was very competitive across various evaluation metrics, even when compared with much bigger models. The following table shows the performance of the base model before being finetuned for story telling:

Nowadays, much stronger models, like Kayra, Erato and Xialong are available on NovelAI, so before long, Clio will get to enjoy a peaceful retirement.

So, in anticipation of this, for the sake of nostalgia, posterity, and historic preservation, we are releasing the weights of our Clio model and her modules publicly on Huggingface Hub under the GPL-2.0 (not "or later") license.

We managed to find an existing model class inside Huggingface Transformers which fits the shape of our architecture for Clio, at least once an additional config flag is set, so no custom code is needed to run Clio. Model files in GGUF format are provided as well.

You can find your very own Clio right here where you are!