LFM2-Research
A small causal language model pre-trained on arXiv AI/ML research papers.
Overview
LFM2-Research is a compact language model built on the LFM2 (Liquid Foundation Model 2) -- i got the code of the model from src/transformers/models/lfm2, architecture and pre-trained exclusively on AI and machine learning research papers from arXiv. It is designed as a lightweight research assistant capable of engaging with technical literature in the AI/ML domain.
⚠️ This model has not undergone RLHF, instruction tuning, or any alignment procedure. It is a raw pre-trained model and is best suited for experimentation and research purposes.
Model Architecture
| Parameter | Value |
|---|---|
| Model Type | Causal Language Model |
| Architecture | LFM2 |
| Hidden Size | 512 |
| Layers | 8 |
| Attention Heads | 8 |
| KV Heads | 4 (Grouped Query Attention) |
| Max Sequence Length | 2048 |
| Vocabulary Size | 50,257 |
Training Details
| Parameter | Value |
|---|---|
| Dataset | FlameF0X/arXiv-AI-ML |
| Training Samples | 2,500 |
| Batch Size | 4 |
| Learning Rate | 3e-4 |
| Epochs | 20 |
| Final Loss | 0.3772 |
Intended Use
- Exploring AI/ML concepts in a research context
- Prototyping lightweight domain-specific language model pipelines
- Studying the effect of narrow-domain pre-training on small models
Out-of-Scope Use
This model is not intended for production use, general-purpose chat, or any application requiring safe, aligned, or factually reliable outputs. It has a very small training set (2,500 samples) and may produce repetitive, incoherent, or factually incorrect text.
Limitations
- Small training set: Only 2,500 samples were used for pre-training, which significantly limits generalization, but because they are research papers are are quite rich.
- No alignment: The model has not been fine-tuned with human feedback or instruction tuning of any kind.
- Potential overfitting: Given the high number of epochs (20) relative to the dataset size, the model may have overfit to training examples.
- Narrow domain: The model has only been exposed to AI/ML research text and will likely perform poorly on out-of-domain inputs.
Safety
Because training data consists solely of academic research papers, the risk of harmful content generation is low. However, the lack of any alignment procedure means outputs are unpredictable and should not be treated as authoritative or safe for end-user-facing applications.
Citation
If you use this model in your work, please cite it as:
@misc{lfm2-research,
author = {FlameF0X},
title = {LFM2-Research: A Small Language Model Pre-trained on arXiv AI/ML Papers},
year = {2025},
url = {https://huggingface.co/FlameF0X/LFM2-Research}
}
Or dont do it. It's up to you.