Boris-2-Preview-1

An early checkpoint of Boris-2, a 144M-parameter base language model from OpenCerebral, trained from scratch on a single NVIDIA V100. This preview is the main run's 50B-token checkpoint with a short learning-rate decay (2.5B tokens) applied. The full run continues to 200B tokens.

This is a base model. It continues text; it does not follow instructions or chat.

Results

Zero-shot, on the float32 weights in this repository: lm-evaluation-harness 0.4.12 for the first four, the leaderboard's own bencharithmark-3.py (unmodified, default settings) for ArithMark-3.

The Index is the Open SLM Leaderboard formula: (N(HellaSwag,25) + N(mean(ARC-E, ARC-C),25) + N(PIQA,50) + 0.65 * N(ArithMark-3,25)) / 3.65, with N(v,c) = 100(v-c)/(100-c).

Single-checkpoint scores on these benchmarks move by roughly plus or minus 0.8 Index between neighbouring checkpoints. No checkpoint or seed was selected on these scores: the rule was set before the decay ran, and this is the one checkpoint at its end. The decay data mix was informed by six 40M-token pilot decays on an earlier run, scored on these same benchmarks; the arms differed by less than single-checkpoint noise, and the mix used here combines them rather than taking the top arm.

With ArithMark-3 run in float32 instead of the script's bfloat16 default it scores 37.00, and the Index is 27.80.

For context, using the Open SLM Leaderboard's published figures for every other model:

The decay moved the Index from 25.10 at the 50B checkpoint to 27.91. Most of the gain is HellaSwag (+5.4) and ARC-Easy (+2.7). ArithMark-3 did not improve: the math data in the decay mix did not transfer to its question format, and the full run's decay will be built around that.

Model

Two components differ from a standard decoder:

• Canon layers. Each block applies a causal 3-tap depthwise convolution to the fused query/key/value projection, so every position mixes in the two previous positions before attention.
• N-gram table. An exact lookup on the last two and three token ids (210,938 bigrams and 23,437 trigrams, no hashing) retrieves a learned rank-64 vector that is added to the residual stream before block 2. It is a gather, not a matrix multiply.

Training packed several documents per sequence with attention, Canon and the n-gram lookup all masked at document boundaries. The modeling code reproduces this: a new document starts after every end-of-text token (eos_resets_context in the config).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "opencerebral/Boris-2-Preview-1"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

x = tok("The water cycle begins when", return_tensors="pt")
y = model.generate(**x, max_new_tokens=40, do_sample=False)
print(tok.decode(y[0], skip_special_tokens=True))

The custom code is plain PyTorch with no extra dependencies. It has no KV cache, so generation recomputes the prefix every step; at this size that is fast enough for sampling and evaluation. Run in float32 or bfloat16. The residual stream is kept in float32 internally because one channel reaches values near the float16 limit.

The converted model was checked against the training code on the same weights: logits match to within 4e-5 in float32, and lm-evaluation-harness reproduces the training evaluator's benchmark scores to within one item.

Training

Main run data, by tokens: about 89% ClimbMix and 11% FineMath.

Decay data, by tokens:

Contamination

No benchmark training, validation or test split was trained on.

• FineMath, OpenMathInstruct-2, Cosmopedia math, Cosmopedia tutorials and DCLM were filtered against every evaluation item with a 13-gram overlap check. The tutorial set had two matching documents out of 179,191 (generic phrases about credit reports and power buttons); both were removed.
• ClimbMix, the bulk of the data, has not been filtered.
• About two thirds of HellaSwag is built from WikiHow articles, and the Cosmopedia tutorials were generated from WikiHow titles. To check whether the HellaSwag gain comes from those topics, the validation items were split by whether their WikiHow article title is also one of the 177,353 tutorial titles (a loose match that puts 43.6% of titled WikiHow items in the overlapping group), and the 50B checkpoint was compared with this one:

The two WikiHow groups started level and both gained about 7 to 8 points. The overlapping group gained 1.1 points more, which is within noise (about 1.4 standard errors); halfway through the decay the order was the other way round (+5.53 against +5.71). The gain is a model getting better at how-to text in general, with at most a small topic effect. ActivityNet items, which have nothing to do with WikiHow, moved much less.

Limitations

A 144M-parameter base model trained on 52.5B tokens. It makes factual and arithmetic mistakes, loses track of longer arguments, and repeats itself under greedy decoding. It has no safety tuning. It is a research preview, not a product.

Sample outputs

Greedy decoding, float32, unedited, cut for length. These are the same twelve prompts the cagliostro-v3.5 card uses; all twelve are in samples.json.

The misses are typical for this size: 47 + 68 comes out as 101, the train question turns into a malformed multiple-choice list, France's capital takes a detour before reaching Paris, and greedy decoding loops on the last prompt.