OELLM 9B — 256K context — SFT

This model is a full-parameter supervised fine-tune (SFT) of birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b on allenai/Dolci-Instruct-SFT, the instruction mixture that Ai2 used for Olmo 3 7B Instruct SFT. It was trained for 2 epochs with sequences of up to 32,768 tokens.

This is an experimental research checkpoint. It has had SFT only, with no preference tuning and no reinforcement learning.

Chat format

The model uses the ChatML-style turn format of Qwen3. The tokenizer is the OpenEuroLLM 256K tokenizer from its qwen3-tokens branch. That branch gives Qwen3 names to eight token IDs that the base tokenizer had reserved:

Unchanged special tokens: <bos> = 1, <eos> = 2, <tool_call> / </tool_call> = 11 / 12, <pad> = 262,144. All other token IDs are identical to the base tokenizer. On a sample of 3.9M tokens of Dolci text, both tokenizers produced identical token IDs.

A rendered conversation looks like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Finland?<|im_end|>
<|im_start|>assistant
<think>

</think>

The capital of Finland is Helsinki.<|im_end|>

• Each assistant turn ends with <|im_end|>. This is the token that SFT taught the model to emit. The pretraining terminator <eos> does not occur in the SFT data. Use <|im_end|> as the stop token.
• The model emits an empty <think> block. The template put a <think>\n…\n</think>\n\n block at the start of the final assistant turn of every conversation. Dolci Instruct has almost no reasoning traces (385 of 2,152,112 conversations have </think> in an assistant message), so the block was nearly always empty. This is not a reasoning model.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,  # puts the empty <think> block in the prompt, as in training
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

generation_config.json lists <|im_end|> (4) and <eos> (2) as stop tokens, so generate() stops at the end of the assistant turn.

Training data

The model was trained on all 2,152,112 conversations of the train split of allenai/Dolci-Instruct-SFT (revision main). Most of the data is English. The dataset card lists the sources and their licenses. The dataset is licensed under ODC-BY and is intended for research and educational use under Ai2's Responsible Use Guidelines.

Before training, the pipeline removed each conversation that had no supervised assistant token in its first 32,768 tokens. It kept conversations where the 32,768-token limit cut a supervised span.

Training procedure

Full-parameter SFT with TRL SFTTrainer.

Loss masking. The loss covers only the assistant tokens that come after the last user message in each conversation, including their closing <|im_end|>. System, user, and tool tokens, and all assistant turns before the last user message, are context only. In a multi-turn conversation, only the final assistant reply (or the chain of assistant turns after the last user message) is trained.

Packing. Conversations were packed with TRL's best-fit-decreasing (BFD) strategy into sequences of up to 32,768 tokens, one packed sequence per device per step. If a conversation was longer than 32,768 tokens, only its first 32,768 tokens were used.

Compute

Reproducibility

Citation

The SFT data comes from the Olmo 3 release:

@misc{olmo2025olmo3,
  title={Olmo 3},
  author={Team Olmo},
  year={2025},
  eprint={2512.13961},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2512.13961},
}

Acknowledgements

Training ran on the LUMI supercomputer, hosted by CSC in Finland.