lfm2.5-230m-ultrachat
A small chat model: LiquidAI/LFM2.5-230M-Base fine-tuned with LoRA on 6,000 examples from UltraChat, then merged into a standalone model. It is an experiment in how far a very short, cheap fine-tune of a 230M-parameter base model goes. It is not a reliable source of facts.
A LoRA-adapter-only version is available at sneezejay/lfm2.5-230m-ultrachat-lora.
Model Details
- Developed by: sneezejay
- Model type: Causal language model (decoder-only), ~230M parameters (229.7M)
- Language(s): English
- License: Inherits the license of the base model. See the base model card for its terms, and note the training-data caveat below.
- Finetuned from: LiquidAI/LFM2.5-230M-Base
- Chat format: ChatML-style template from the base tokenizer (
<|im_start|>user ... <|im_end|>). Usetokenizer.apply_chat_template.
Uses
Direct use
- Experiments with small-model fine-tuning, chat formatting, and quantization or conversion pipelines.
- A lightweight baseline to compare other small fine-tunes against.
- Short, casual question-and-answer chat where factual accuracy is not important.
Out-of-scope use
- Anything that depends on correct facts: medical, legal, financial, safety, or academic use.
- Code generation you intend to run without review.
- Production systems, or any setting where a confident wrong answer could cause harm.
Bias, Risks, and Limitations
- It makes things up. In informal testing it wrote fluent, well-organized answers that contained clear errors. For example, when asked how viruses differ from bacteria, it gave wrong claims about replication and cell structure, and its Python dictionary explanation invented details. This is typical of a 230M model and a short fine-tune.
- No fixed identity. The training data does not teach the model who it is, so answers to "who are you?" are generic and inconsistent.
- Training data is synthetic. UltraChat conversations were generated by another language model, so the model inherits that model's style, mistakes, and biases (verbosity, a generic "assistant" tone, Western-centric framing).
- Limited training. Only 6,000 examples and one pass, so the model has seen very little. It may repeat itself, drift off topic, or cut off mid-sentence on longer outputs.
- Not safety-tuned. No dedicated safety or refusal training was done beyond whatever is in the base model and the data. Do not rely on it to refuse harmful requests.
Recommendations
Treat outputs as untrusted. Verify any factual claim or code independently. If you use or redistribute this model, review the licenses of the base model and of the UltraChat dataset (which is itself built from model-generated text) for your use case.
How to Get Started
from transformers import pipeline
pipe = pipeline("text-generation", model="sneezejay/lfm2.5-230m-ultrachat")
messages = [{"role": "user", "content": "Give me three tips for staying focused while studying."}]
out = pipe(messages, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(out[0]["generated_text"][-1]["content"])
Suggested sampling: temperature 0.7, top_p 0.9, repetition_penalty ~1.1.
Training Details
Training data
HuggingFaceH4/ultrachat_200k, train_sft split. 6,000 conversations were chosen at random (shuffle seed 42) from the 207,865 available. Multi-turn chat data in messages format.
Preprocessing
Each conversation was rendered to text with the base tokenizer's chat template and truncated to 2,048 tokens. No other filtering was applied.
Training procedure
- Method: Supervised fine-tuning with LoRA (PEFT) using TRL's
SFTTrainer, then the adapter was merged into the base weights. - LoRA: rank 64, alpha 128, dropout 0.05, applied to all linear layers (about 15.5M trainable parameters, ~6.3% of the total).
- Epochs: 1 (planned 375 optimizer steps; the final adapter was saved from the in-memory weights at about step 350 after a CUDA out-of-memory error near the end of the run, when the learning rate had almost fully decayed).
- Batch size: 4 per device, 4 gradient accumulation steps (effective batch 16).
- Learning rate: 2e-4, cosine schedule, 11 warmup steps, weight decay 0.01.
- Optimizer: AdamW (
adamw_torch). - Precision: fp32 weights with fp16 mixed precision.
- Sequence length: 2,048 tokens, no packing, loss computed over the whole formatted conversation (not assistant-only).
- Training loss: about 1.73 at step 20, settling around 1.55 to 1.6 by the end.
Speeds, sizes, times
About 38 minutes for ~350 steps on a single NVIDIA Tesla T4 (14.6 GB).
Evaluation
No benchmark evaluation was run. The model was only spot-checked on a few prompts (explaining a Python dictionary, study tips, virus vs. bacterium, "who are you?"). It reliably follows the chat format and writes organized answers, but factual accuracy is poor (see Limitations).
Environmental Impact
- Hardware: 1x NVIDIA Tesla T4
- Hours used: ~0.65 (training only)
- Cloud provider: Google Colab
- Compute region: Not recorded
- Carbon emitted: Not measured
Technical Specifications
- Architecture: LFM2.5 (Liquid Foundation Model) hybrid architecture, 230M parameters. See the base model card for details.
- Objective: Next-token prediction (causal language modeling) on chat-formatted text.
- Software: Hugging Face Transformers, PEFT, TRL, Datasets, PyTorch (exact versions not recorded).
Model Card Authors
sneezejay