rost-1b-instruct-v2

A 1.384B-parameter bilingual Romanian/English instruction model. v2 replaces the supervised fine-tuning mixture behind rost-1b-instruct. The pretrained base is unchanged, so anything the base knows, both versions know equally.

v2 materially reduced token-cap failures under the internal benchmark and a six-prompt repetition-stress protocol, but did not eliminate looping completely.

It also improved instruction-following on the internal suite, from 12/75 to 19/75. It did not improve public likelihood benchmarks — it is slightly behind v1 there — and the sealed 44-case held-out evaluation did not confirm a general improvement over v1. All of that is below.

What changed

v2 is arm C of a three-arm supervised fine-tuning experiment. The arms shared a frozen batch plan, identical optimizer settings and identical behaviour-shaping rows, and differed only in mixture composition; arm C added mathematics and a code-persona source. Selection was made on public suite 1.2 deterministic task success plus a held-back probe set, and frozen before any held-out evaluation was opened.

Decoding

This repository sets only the required token IDs and prescribes no sampling parameters.

No tested decoding configuration satisfied the preregistered requirement of zero looping and zero token-cap endings on all six repetition-stress prompts. This repository therefore sets only the required token IDs and does not prescribe sampling parameters. Transformers defaults to greedy decoding unless callers provide their own settings. Under greedy decoding, v2 reached its stop token on all six prompts but exhibited looping on the "seasons" prompt.

Greedy is not an endorsement. It did not pass the gate either; it is what prescribing nothing leaves, and prescribing nothing is the least assumptive response to having measured nothing that worked.

generation_config.json in full:

{
  "_from_model_config": false,
  "bos_token_id": 32759,
  "eos_token_id": [32763, 32759],
  "pad_token_id": 32763
}

eos_token_id lists both <|assistant_end|> and <|bos|> because the model emits either to end a turn. Without an eos_token_id at all, generate() has no stop condition and runs to max_new_tokens on every call.

Termination and repetition, measured

Six prompts chosen to provoke looping, five decoding settings, 400 tokens maximum. Greedy is deterministic and reports six outputs, one per prompt; the stochastic settings are 48 draws each, clustered eight per prompt.

Generations that ran out of budget instead of stopping:

Looping, by prompt rather than by rate. Under greedy, v1 loops on three of the six prompts (essay, opposite, seasons); v2 loops on one, seasons — and seasons is the only prompt v2 loops on in every configuration tested. The aggregate rate would read as a mild general tendency; it is one input.

In this small, prompt-clustered stress test, increasing the penalty from 1.1 to 1.15 did not improve repetition and produced more classified loops.

Six adversarial prompts cannot establish that termination is fixed in general. The claim above is scoped to this protocol and the internal benchmark, and to nothing else.

Evaluation

Four independent measurements. They do not all point the same way, and the disagreement is informative rather than embarrassing: v2 changed assistant behaviour, and only one of these four measures behaviour.

1. Internal suite 1.2 — instruction following

75 active cases, scorer 1.3.0, paired_deterministic (greedy, seed 42, 128 new tokens, repetition penalty 1.1), code execution required, no CLI overrides, offline, both models loaded from their published revisions.

Per case: 10 both pass, 9 v2 only, 2 v1 only, 54 both fail. A sign test on the 11 non-ties gives p ≈ 0.065. Termination on the same run: v2 completes on a stop token 33/75 against v1's 15/75.

2. Public English likelihood battery

lm_eval==0.4.12 pinned, transformers 5.15.1, torch 2.9.1. Nine tasks zero-shot, MMLU five-shot. One metric per task, fixed before any result was read, applied identically to every model. Peer models were measured under this same harness rather than quoted from their own cards, because published cards use different harnesses at different shot counts and mixing them would make the columns incomparable.

Metric per task: ARC-C acc_norm, ARC-E acc, BoolQ acc, HellaSwag acc_norm, LAMBADA acc, OpenBookQA acc_norm, PIQA acc, SciQ acc, WinoGrande acc, MMLU acc. The macro mean is a convenience and is arbitrary in the usual way — it weights a nine-point spread on SciQ like one on MMLU. The per-task values are the record.

v2 is slightly below v1 here, winning two of nine tasks. This is a small post-training likelihood regression: one supervised fine-tuning recipe lowering option likelihood slightly on one battery. It is not evidence of a general mechanism.

The peer rows are context, not a target. RoST's base saw 11.68B training tokens, single pass; both peers saw far more.

3. Public Romanian battery

Five OpenLLM-Ro tasks, zero-shot, 400 rows per task, likelihood-scored, identical cached rows for both models.

At n=400 the per-task standard error is about ±2.4 points, so every delta including the mean is inside noise. No measurable difference. Read these against the chance column: both versions are near chance on ro_arc_challenge and ro_winogrande.

These figures are not comparable to the Romanian table on v1's card, which reports the base checkpoint. This is a new instruct-versus-instruct comparison with no published baseline.

4. Sealed held-out suite

A 44-case suite was written and encrypted before training, held by an independent reviewer, and decrypted exactly once after the winner was frozen. It selected nothing; it existed only to test whether the improvement generalises.

It did not confirm that it does. The two measures disagree in sign and neither is strong:

31 of 44 cases scored zero for both models, so the discriminating set is eight cases at most; an exact sign test on the non-ties gives p ≈ 0.73. The comparison was also heavily confounded by that suite's frozen 128-token cap, which truncated 34 of v1's generations and 25 of v2's. The honest reading: the held-out test did not establish that v2 generalises better — and equally did not establish that v1 is better.

The suite is consumed and will not be rerun.

Limitations

• Daily conversational quality is unmeasured. No benchmark here is a proxy for it.
• The seasons prompt loops under every decoding setting tested. Asking for an enumeration and explanation of all four seasons is a known failure.
• Hallucination resistance did not improve — 0/4 on both public suites.
• Coding shows no convincing improvement.
• It is a 1.384B model. Arithmetic and multi-step reasoning are wrong routinely.
• 4,096 tokens of context.
• No safety tuning of any kind has been applied.
• Several internal cases are scored by exact match against the first non-empty line, so a correct answer inside a longer sentence does not count.

Provenance

Revisions

Every number on this page was produced from d833f89f. The final card revision is a later commit that changes only this file; all behaviour-affecting files — weights, config, tokenizer, modelling code, generation config — are byte-identical between the two, and that identity is verified from a clean download rather than asserted. The card revision was not itself benchmarked, and nothing here should be read as implying it was.

An earlier revision 2250a44c carried the same weights without generation_config.json, the raw checkpoint or the nanochat tokenizer artifacts. It is superseded.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "rostlabs/rost-1b-instruct-v2"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()

messages = [{"role": "user", "content": "Care este capitala României?"}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(inputs["input_ids"].to("cuda"), max_new_tokens=128)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The architecture is not a Llama or Mistral derivative, so the modelling code ships in this repository and trust_remote_code=True is required.

The original training checkpoint (model_000365.pt, meta_000365.json) ships alongside the safetensors, for use with the training code.

Licence

CC-BY-NC-4.0, non-commercial. The model inherits non-commercial terms from both halves of its training data: the base was pretrained on Nemotron ClimbMix (CC-BY-NC-4.0), and the v2 supervised mixture includes OpenLLM-Ro/ro_sft_ultrachat and OpenLLM-Ro/ro_gsm8k, both CC-BY-NC-4.0. Other sources in the mixture are more permissive — OpenMathInstruct-2 (CC-BY-4.0), smol-smoltalk (Apache-2.0), tulu-3-sft-mixture (ODC-BY-1.0 / Apache-2.0) — but the most restrictive term governs. The Romanian pretraining data is ODC-BY and requires attribution to FineWeb2.

Citation

@misc{rost2026,
  title  = {rost: a bilingual Romanian-English language model trained from scratch},
  author = {Iancu, Stefan},
  year   = {2026},
  url    = {https://huggingface.co/rostlabs/rost-1b-instruct-v2}
}