Sarath569/slm-125m-legal-chat
An instruction-tuned ~125.8M Llama-style model for legal / financial text. Supervised fine-tuned from a from-scratch base (Sarath569/slm-125m-legal-base, further pretrained to 4 epochs over ~2.04B tokens of US case law + SEC filings + educational web; base val perplexity 9.66) on a blend of general and domain instructions.
Chat format
The model uses the reserved chat tokens. Format each turn as:
<|user|>
{your question}
<|assistant|>
{model answer}<|eos|>
SFT data (33,601 examples, 2 epochs)
databricks/databricks-dolly-15k— general instructions (~15k)gbharti/finance-alpaca— financial Q&A (~15k)dzunggg/legal-qa-v1— US legal Q&A (~3.7k)
Loss on assistant-response tokens only; LR 2e-5 cosine. Sensible decoding defaults
(repetition_penalty=1.3, no_repeat_ngram_size=3, temperature=0.7) are baked into
generation_config.json.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Sarath569/slm-125m-legal-chat")
model = AutoModelForCausalLM.from_pretrained("Sarath569/slm-125m-legal-chat")
prompt = "<|user|>\nExplain in simple terms what a 10-K filing is.\n<|assistant|>\n"
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160) # decoding defaults come from generation_config
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Limitations (please read)
This is a ~125M model. After SFT it follows the chat format and produces fluent, legal/financial-flavored answers and stops appropriately, but it is not a reliable source of legal or financial facts — it hallucinates, and must not be used for legal, financial, or any real-world advice. It is best on general/common topics and weakest on precise domain facts. It is a base+SFT model with no alignment/RLHF. Intended for research and as a small fine-tuning starting point, not production use.