slm-125m-base

asinha08/slm-125m-base is a 125M parameter, Llama-compatible, decoder-only causal language model trained from scratch on a legal-first English corpus. It is a base language model, not an instruction-tuned or chat-aligned assistant.

Model Details

Special tokens:

The extra chat-style tokens are present in the tokenizer, but the model was not instruction tuned.

Intended Use

This model is intended for:

• Research on small language model pretraining.
• Legal-domain language modeling experiments.
• Continued pretraining, supervised fine-tuning, or evaluation pipelines.
• Text continuation/completion tests.

This model is not intended for:

• Legal advice or compliance decisions.
• Production legal automation without task-specific evaluation and guardrails.
• Chatbot use without instruction tuning and safety tuning.
• Factual question answering where accuracy is required.

Training Data

The corpus was built from three public Hugging Face datasets with a legal-first mix:

Data processing included:

• Basic line/document cleaning.
• OCR/noise filtering for case-law data.
• Exact duplicate removal.
• MinHash near-duplicate removal.
• Decontamination against held-out evaluation sources: coastalcph/lex_glue and casehold/casehold.
• Byte-level BPE tokenizer training on the cleaned corpus.
• Packed fixed-length token windows with sequence length 1,024.
• Validation routing of every 100th packed window, giving an approximately 99/1 train/validation split.

Deduplication and decontamination summary:

Training Procedure

Training ran on Modal using 8x H100 GPUs.

The model has completed 4 full training epochs over the packed train split.

Evaluation

Perplexity is computed as exp(validation_loss).

The reported validation metrics use 4,096 validation windows, or about 4.19M validation tokens, sampled from the internal packed validation split. The full validation split contains 20,119 windows, or 20,601,856 tokens.

The latest completed evaluation is:

No downstream task benchmark results are reported yet.

Approximate Training Cost

Known pretraining cost by epoch:

Approximate known pretraining total: ~$39.19+. This excludes any unrecorded dataset preparation, storage, network, or earlier CPU-only costs.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "asinha08/slm-125m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "<|bos|>The plaintiff alleged that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=False))

Because this is a base model, prompts should be treated as text-completion prefixes. The tokenizer contains <|user|>, <|assistant|>, and <|system|> tokens, but the model has not been trained to follow chat instructions.

Limitations and Risks

• The model is small and undertrained relative to modern production LLMs.
• The model can hallucinate, fabricate citations, and produce legally incorrect text.
• The model is not safe for legal advice, compliance review, contract review, court filings, or any high-stakes use without extensive validation.
• The model was not instruction tuned, RLHF tuned, preference tuned, or safety aligned.
• The training corpus is mostly legal/financial English text plus a smaller web-education slice; performance outside that distribution may be poor.
• The model may reproduce biases, sensitive patterns, or memorized fragments from public data.
• Evaluation currently reports only internal validation perplexity, not downstream legal-task accuracy.
• Source datasets have their own licenses and use constraints; users should verify dataset licenses before commercial or redistributed use.

Reproducibility Notes

Main artifacts were stored on the Modal volume slm125mLIVE:

Latest training summary:

Citation

No formal citation is available. If you use this model, cite the Hugging Face repository:

asinha08/slm-125m-base