CRIA-LM-75M-Instruct

Cria (noun): a baby llama, alpaca, vicuña, or guanaco. Pronounced ˈkrē-ə.

~ Merriam-Webster

cRia-LM-75M-Instruct is a 75.7M-parameter instruction-tuned language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal.

The model starts from cRia-LM-75M, then adds supervised instruction tuning and preference optimization. It ships with a native chat template using <|im_start|> and <|im_end|> markers.

Model Details

Architecture

cRia-LM-75M-Instruct keeps the base model's 13 unique Transformer layers:

1 prelude + (11 shared layers x 2 recurrent passes) + 1 coda

This gives an effective depth of 24 Transformer layers while storing the main parameters for 13 unique layers. The recurrent block shares its base weights across both passes. Rank-172 LoRA updates provide separate capacity on the second pass.

The full architecture and base-model history are documented on the cRia-LM-75M model card.

Training

Post-training starts from cRia-LM-75M. Base-model training details are kept on that model card.

Supervised fine-tuning

The first part uses the full training split of HuggingFaceTB/smol-smoltalk. Only assistant turns contribute to the language-model loss.

HuggingFaceTB/SmolLM2-360M-Instruct supplies same-tokenizer KL targets during the first SFT epoch. The teacher temperature is 1.2. A lagged CE-to-KL ratio sets the distillation scale after 50 calibration steps, ramps it over 250 steps, and caps it at 0.5 so ground-truth cross-entropy remains the main objective.

Muon updates the ordinary two-dimensional Transformer matrices. AdamW handles embeddings, norms, recurrent LoRA and QK-Norm parameters, and the residual-gain scalars.

Direct Preference Optimization

The SFT checkpoint is then trained on the train_prefs split of HuggingFaceH4/ultrafeedback_binarized.

Reference log probabilities are precomputed from the SFT checkpoint. Training uses BF16 computation with TF32 matrix multiplication enabled.

Evaluation

The tables below report cRia-LM-75M-Instruct at revision 57c83387f67b8c808b9f4f18c1c0047b72f66e24.

All runs used complete evaluation sets with no sample limits. Model tensors and log-softmax calculations used FP32, CUDA matrix multiplication used TF32, and the shared evaluation cap was 2,048 tokens. SmolLM and SmolLM2 were evaluated under the same local protocol. These numbers were measured for this card and were not copied from their model cards.

Instruction benchmarks

IFEval and BBH use each checkpoint's native chat template. BBH is zero-shot. MT-Bench uses the official 80-question, two-turn set with single-answer grading by gpt-5.6-luna.

Basic text benchmarks

HellaSwag, ARC-Easy, ARC-Challenge, and PIQA use zero-shot normalized continuation likelihood with each model's native chat template. ArithMark-3 uses its raw-text continuation protocol at a 1,024-token context cap.

These scores compare checkpoints under one fixed local setup. cRia has 75.7M parameters, while both comparison models have about 135M. Prompt format, harness version, precision, and shot count can move small-model scores by several points.

Usage

The model ships with a custom Transformers implementation, so trust_remote_code=True is required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sz14/cRia-LM-75M-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device).eval()

messages = [
    {"role": "system", "content": "Answer clearly and concisely."},
    {"role": "user", "content": "Explain why the sky is blue."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
    )

new_tokens = output[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The chat template inserts a default system message when one is not supplied. Pass a system message explicitly when an application needs consistent behavior.

Intended Use

cRia-LM-75M-Instruct is intended primarily for:

• research on compact instruction-tuned models
• experiments with recursive parameter sharing
• local prototypes with limited memory
• further supervised fine-tuning and preference optimization
• testing small-model chat and tool pipelines

Limitations

cRia-LM-75M-Instruct is a very small language model. Its responses can be short, repetitive, incorrect, or poorly formatted.

Known limitations include:

• weak multi-step reasoning
• limited factual knowledge
• low instruction-following reliability on complex prompts
• substantial quality loss across multi-turn conversations
• potential factual errors and hallucinations
• English-focused training
• a 4,096-token maximum context window

The model should not be treated as a reliable source of factual information or used without additional validation in high-stakes applications.

Architecture and Training Lineage

The Transformer layer design follows HuggingFaceTB/SmolLM2-135M, released by Hugging Face under the Apache 2.0 license.

The recursive parameter-sharing approach is based on:

Bae et al., "Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA," arXiv:2410.20672.

The SFT distillation scale follows the pooled, lagged balancing idea used in cross-tokenizer distillation work:

Sreenivas et al., "X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation," arXiv:2605.21699.

License

cRia-LM-75M-Instruct is released under the Apache License 2.0.