NovelAI-LM-3B-508k (Clio's base model)
This is the model that Clio is based on. It was trained on our Nerdstash dataset with a context size of 2048 tokens. It is not finetuned specifically for story telling, so it might serve as a useful basis for other tasks. At the time of Clio's release, this base model was SOTA for its size according to our evaluation metrics. Comparing it to more recent completion models is more difficult, because the common set of evaluation metrics has changed.
We managed to find an existing model class inside Huggingface Transformers which fits the shape of our architecture, at least once an additional config flag is set, so no custom code is needed to run the model. Model files in GGUF format are provided as well.