ARC-Encoder models
This page houses ARC8-Encoder_multi from three different versions of pretrained ARC-Encoders. Architectures and methods to train them are described in the paper ARC-Encoder: learning compressed text representations for large language models available here.
Code: ARC-Encoder repository
Models Details
All the encoders released here are trained on web crawl filtered using Dactory based on a Llama3.2-3B base backbone. It consists in two ARC-Encoder specifically trained for one decoder and one for two decoders in the same time:
ARC8-Encoder_Llama, trained on 2.6B tokens on Llama3.1-8B base specifically with a pooling factor of 8.ARC8-Encoder_Mistral, trained on 2.6B tokens on Mistral-7B base specifically with a pooling factor of 8.ARC8-Encoder_multi, trained by sampling among the two decoders with a pooling factor of 8.
Uses
As described in the paper, the pretrained ARC-Encoders can be fine-tuned to perform various downstream tasks. You can also adapt an ARC-Encoder to a new pooling factor (PF) by fine-tuning it on the desired PF. For optimal results, we recommend fine-tuning toward a lower PF than the one used during pretraining. To reproduce the results presented in the paper, you can use our released fine-tuning dataset, ARC_finetuning.
Licensing
ARC-Encoders are licensed under the CC-BY 4.0 license.
Terms of use: As the released models are pretrained from Llama3.2 3B backbone, ARC-Encoders are subject to the Llama Terms of Use found at Llama license.
Usage
To load the pre-trained ARC-Encoders, use the following code snippet from the ARC-Encoder repository:
from embed_llm.models.augmented_model import load_and_save_released_models
# ARC8_Encoder_multi, ARC8_Encoder_Llama or ARC8_Encoder_Mistral
load_and_save_released_models(ARC8_Encoder_multi, hf_token=<HF_TOKEN>)
Remark: This code snippet loads the model from Hugging Face and then creates appropriate folders at <TMP_PATH> containing the checkpoint and additional necessary files for fine-tuning or evaluation with the ARC-Encoder codebase. To reduce occupied memory space, you can then delete the model from your Hugging Face cache.
Citations
If you use one of these models, please cite:
@article{
pilchen2026arcencoder,
title={{ARC}-Encoder: learning compressed text representations for large language models},
author={Hippolyte Pilchen and Edouard Grave and Patrick Perez},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2026},
url={https://openreview.net/forum?id=lU1P9dsqfn},
note={Featured Certification}
}