PasDePitiéPourLeCroissantLLM - Base (demo v20240519-7h18)

PasDePitiéPourLeCroissantLLM was created by UnBias(https://www.unbias.fr) with its propriatory Harpax technology from the CroissantLLM initiative's 24 layers, 1.345 (a.k.a "1.5") billion parameters croissantllm/CroissantLLMBase model (5,3 Go ROM) to 282 million parameters and 3 layers (709 Mo ROM).

That's a whopping 80% reduction in size, speed, energy and carbon consumption. Your 100,000 euros compute budget could either afford you 5x the compute or you could save 80.000 euros.

This version of the model is however NOT ENABLED with UnBias Harpax CrystALS as it has been reverted to a vanilla architecture by UnBias Harpax UnCrystALS. The CrystALS version is 182 millions parameters and fits on 200Mo ROM.

A UnBias Harpax CrystALS version would result in a 87% reduction compared to baseline in size, speed, energy and carbon consumption. Your 100,000 euros compute budget could either afford you 7.7x the compute or you could save 87.000 euros.

Last but not least the larger the model gets, the larger the compression factor becomes, about 50x-ish for 70billions parameters-class models.

To harness the benefits of UnBias Harpax CrystALS please liaise with our sales department for further enquiries.

Pas de Pitié pour les Croissants ("Spare no Croissants") is a pun on CroissantLLM based on a 1980s French children television broadcast (© 1987, AB Productions, TF1)

Abstract

Who we are ?

UnBias is a French DeepTech start-up based at Sophia-Antipolis focusing on high-precision, portable (Mb not Tb) signal understanding and bias control.

At UnBias, we strive for sustainability and efficiency in furthering artificial intelligence and machine learning. Our frugal R&D regularly yields discoveries that accelerate model training while using minimal resources.

What we do?

UnBias Harpax comprises 4 components to elevate your efficiency and win the AI race:

1. Trainer, the acceleration component
2. CrystALS, the compression architecture
3. Vampire the third-party non-native model conversion to CrystALS architecture
4. UnCrystALS module to export back to non-CrystALS architectures for sharing and publications (provided a loss in performance and increased size)

UnBias Harpax was designed for large-scale pre-training from scratch, eventually we extended the solution to vampirize existing models

What is this?

PasDePitiéPourLeCroissantLLM was created by UnBias Harpax from the CroissantLLM initiative's 24 layers, 1.345 (a.k.a "1.5") billion parameters croissantllm/CroissantLLMBase model (5,3 Go ROM) to 282 million parameters and 3 layers (709 Mo ROM).

Benefits

Our layer supercompression technique shrinks model footprint several times without quantization nor pruning.

With less GPU required, our solution offers significant savings on compute for model both model pre-training from scratch or further-training as well as fine-tuing and inference.

This will save both considerable expenditures in compute, development lead-time and deployment.

Smaller models also fit on smaller, typically 30-40% cheaper GPUs, resulting in considerable potential cost savings.

Last but not least, smaller models may fit on CPUs for added edge and remote serving as well as IoT deployments.

By reducing the model footprint many times, UnBias Harpax CrystALS helps reduce carbon costs and makes it more affordable for businesses to deploy solutions.

Contact us to learn more about how CrystALS can help your research and develop in a responsible and sustainable manner.

CrystALS is a pun on Crystal to pursue the Sesame Street tradition initiated by BERT (© 1984, The Sesame Workshop)

Citation

Our work can be cited as:

@misc{dalferro2024pasdepitiépourlecroissantllm,
      title={PasDePitiéPourLeCroissantLLM: An UnBias Harpax Vampire demo compression of a Truly Bilingual French-English Language Model}, 
      author={{UnBias SAS}}, Benoit Dal Ferro, Daphné Marnat},
      year={2024},
}

The CroissantLLM's Initiative can be cited as:

@misc{faysse2024croissantllm,
      title={CroissantLLM: A Truly Bilingual French-English Language Model}, 
      author={Manuel Faysse and Patrick Fernandes and Nuno M. Guerreiro and António Loison and Duarte M. Alves and Caio Corro and Nicolas Boizard and João Alves and Ricardo Rei and Pedro H. Martins and Antoni Bigata Casademunt and François Yvon and André F. T. Martins and Gautier Viaud and Céline Hudelot and Pierre Colombo},
      year={2024},
      eprint={2402.00786},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Usage

This model is a pre-trained foundation model, that is, it is NOT fine-tuned to perform specific tasks e.g. Chat function. Please fine-tune this model prior to assessing its performance on your use-case.

Supercompressed models exhibit unusual temperature and top_k behaviour please tune carefully.