OELLM 9B — 256K context (ABF, θ=64M) — prelude (1T-token base)
256K long-context extension of the OpenEuroLLM prelude base (Qwen3 dense 9B,
qwen3_9b_hf_baby at iter_0124800 ≈ 1T tokens). Continues the θ-law curriculum from our
128K model (oellm-9b-128k-theta32m-prelude)
to the next octave. Base model — not instruction-tuned. Multilingual (37 European languages).
Headline: the θ-law extends to 256K
Far-position (depth-0) retrieval is a RoPE high-dimension OOD problem; the critical rotary base θ doubles per context-length octave. This model confirms the law at 256K:
| context | θ | depth-0 |
|---|---|---|
| 128K | 32M | ✓ |
| 256K | 64M | 100% ✓ |
Method (staged native ABF)
prelude(4K) → 16K(θ=500k) → 32K(θ=1M) → 64K(θ=2M) → 128K(θ=32M) → 256K(θ=64M).
Continued-pretraining in Megatron-LM on LUMI (16× MI250X), CP=16, θ=64M, selective recompute,
1B tokens for the 256K stage. Data: 60% genuine-256K long documents (arXiv/books/code/RFC/docsite
concatenated to 256K) + 40% multilingual long-context mix (Jouni Luoma's 152-source blend) — the
multilingual portion prevents non-English regression during the 256K stage.
Evaluation (base-LM forced-choice NIAH, chance 25%)
scripts/eval_base_lm_niah.py. The 256K
forward needs multi-GPU (device_map="auto" over a full 8-GCD node; one GCD OOMs at 256K).
@256K (7 of 15 languages, 185 trials; full sweep in progress). Overall 90% (168/185).
| depth | 0.0 | 0.25 | 0.5 | 0.75 | 1.0 |
|---|---|---|---|---|---|
| accuracy | 93% | 86% | 87% | 88% | 100% |
By language: fr 100%, en 98%, pt 96%, pl 93%, de 83%, el 81%, sv 76%. depth-0 = 93% confirms theta=64M solves far-position retrieval at 256K. Mid-depths (0.25/0.5/0.75) in the mid-80s reflect the mild lost-in-the-middle inherited from the 1T base (see 128K-prelude card). 128K retention: 98%. (depth 0.5 = 93%, the mild mid-depth softness carried from the 128K model; see that model's card).
Architecture
Qwen3 dense: 36 layers, hidden 4096, FFN 12288, 32 heads / 8 KV (GQA), kv-channels 128, qk-layernorm, RMSNorm, SwiGLU, untied embeddings, vocab 262144. Config: rope_theta=64000000, max_position_embeddings=262144.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# 256K context needs several GPUs; use device_map="auto".
m = AutoModelForCausalLM.from_pretrained("openeurollm/oellm-9b-256k-theta64m-prelude",
torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("openeurollm/oellm-9b-256k-theta64m-prelude")
# Base completion model. Keep rope_theta=64M / max_position=262144 for 256K.
Caveats
- Base model (no instruction tuning). Single-needle NIAH; broader RULER not yet run.
- Keep
rope_theta=64000000,max_position_embeddings=262144for 256K. - Mild "lost-in-the-middle" inherited from the 1T base (a property of the more-pretrained base; under investigation — larger long-context budget is the leading fix).
- Repo/code: https://github.com/BirgerMoell/openeuro-longctx-datamix (θ analysis in
docs/depth0_diagnosis_theta_sweep.md). Sibling:oellm-9b-128k-theta32m-prelude.