cagliostro-v2-sft
An instruction tuned version of cagliostro-v2, a 150M parameter model trained from scratch on 50B tokens.
Fine tuned on 196,237 conversations from smoltalk, which is built for models at this scale.
What this is and is not
It answers in the right shape. It picks up the conversational format and stays on topic for a sentence or two.
It does not reliably stop. Measured at a natural end of answer, the probability
it assigns to the end of text token is 2 to 7 percent, which puts it second to
eighth in the ranking, so greedy decoding never selects it. In testing, zero of
four prompts terminated on their own. Always pass max_new_tokens, and add a
stopping criterion on "\nUser:" if you need clean turn boundaries.
It is not a useful assistant. At 150M parameters the base model already gets simple facts wrong with complete confidence, and instruction tuning does not change what a model knows, only how it presents what it knows. Expect correct formatting wrapped around unreliable content.
If you want a sense of the ceiling, the base model scores 19.99 on the Open SLM Intelligence Index, where ARC-Challenge sits at 28.58 against a chance floor of 25.00. That is a model with a weak grasp of the world, and fine tuning inherits it.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"bench-labs/cagliostro-v2-sft", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("bench-labs/cagliostro-v2-sft")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
trust_remote_code=True is required. The architecture is not in transformers
and travels with the weights.
Chat format
Plain text markers rather than new special tokens:
User: {question}
Assistant: {answer}<|endoftext|>
The base model has tied embeddings, so adding vocab entries would mean resizing
and then training rows that start as noise. For a run this short that is a bad
trade, and User: and Assistant: tokenize cleanly with the existing
vocabulary. <|endoftext|> is a real token in the vocabulary (id 0) and marks the end of
each assistant turn in the training data, but see the note above: the model did
not learn to emit it confidently enough for greedy decoding to stop.
Training
| Base | bench-labs/cagliostro-v2 |
| Data | HuggingFaceTB/smoltalk, 196,237 conversations |
| Tokens | 117.9M |
| Epochs | 1 |
| Sequence length | 1024 |
| Batch | 4 x 8 accumulation |
| Optimizer | AdamW, betas 0.9/0.95, weight decay 0.01 |
| LR | 5e-5, one cycle cosine, 3 percent warmup |
| Precision | bfloat16 |
| Hardware | 1x RTX 3060 |
Loss is computed on assistant turns only. The user's text is masked out, since training on it teaches the model to write questions rather than answer them.
Limitations
Not aligned, not filtered, not safe for production. It will state false things
confidently, repeat itself on longer generations, and has no refusal behaviour
of any kind. Asked for the capital of France it answers Paris and then places
Paris in the French Riviera. Asked to define a prime number it says a number
divisible by itself. Some prompts trigger <tool_call> output, because smoltalk
contains function calling conversations and one epoch was not enough to learn
when they apply. It exists to make the base model usable for testing conversational
pipelines, not to be talked to.