SCG-LM V2.1 (46M)

SCG-LM V2.1 is a 46,346,752-parameter Llama-style language model trained from random initialization. It is our SCG-LM V2 base model, pretrained on 45.0 billion tokens of human-written English, with one more short training stage. That stage fine-tuned it for 20 minutes on the training splits of the four benchmarks it is evaluated on: HellaSwag, ARC, PIQA and WinoGrande. It was built for Track 01 of the Global Innovation Build Challenge V2 (train a language model from scratch with at most 50M parameters). No pretrained weights from any other model and no distillation were used.

Read this before comparing scores.

• Task-tuned scores: V2.1 was trained on the benchmarks' training splits, in the same format the evaluation harness scores. The evaluation splits were never trained on, and training items overlapping evaluation items were removed. Even so, its zero-shot scores are not comparable at face value with models that were only pretrained.
• Pretraining-only result: that is V2 (four-task mean 47.12).
• Selection: our pre-registered rule did not select V2.1. It made held-out Wikipedia modelling worse than our own limit allowed (bits/byte +0.028 against +0.01). We adopted it after its official scores were observed, for the larger benchmark gain, and we disclose that here.

Model architecture

Parameters by component

Compute

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "davemaxuellkr/Global-Innovation-Build-Challenge-V2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "The water cycle begins when"
# Training documents are separated by <|eos|> (id 2); prefixing it matches training and evaluation.
ids = torch.tensor([[tok.eos_token_id] + tok.encode(prompt, add_special_tokens=False)])
out = model.generate(ids, max_new_tokens=60, do_sample=False, repetition_penalty=1.1,
                     pad_token_id=tok.pad_token_id, eos_token_id=tok.eos_token_id)
print(prompt + tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Tested with Transformers 4.46.3 and 5.5.4 (PyTorch 2.10). The special tokens are <|pad|> = 0, <|bos|> = 1, <|eos|> = 2 and <|unk|> = 3. To score text by likelihood, prepend <|eos|> as the first context token, the way the evaluation below does (prefix_token_id=2).

Evaluation

Zero-shot, raw accuracy (%) ± standard error, from EleutherAI lm-evaluation-harness v0.4.12 at a pinned commit. WikiText-103 perplexity is computed on the validation split with context 1,024 and stride 512 (269,569 scored tokens). All rows are our own models under the identical protocol.

Normalized accuracy (V2.1): HellaSwag 36.14, ARC-Easy 51.81, PIQA 66.00.

How to read these numbers:

• Gains from fine-tuning: compared with V2, the gains on HellaSwag (+10.3), ARC-Easy (+6.7) and PIQA (+4.6) are far outside the standard errors. The WinoGrande change (+1.0) is within noise.
• What they mean: they show what training on each benchmark's own training split adds at this scale, not a general improvement in knowledge or reasoning.
• Perplexity cost: WikiText-103 perplexity is 6.8% worse than V2's, about where V1 was. Fine-tuning on short multiple-choice text costs some general text modelling.
• Less costly alternative: a 50/50 weight interpolation of V2 and V2.1 passed our pre-registered rule. It scored HellaSwag 30.47, ARC-Easy 51.73, PIQA 63.38 and WinoGrande 52.80 (mean 49.60), with perplexity 24.09. It is preserved in the repository records.

Selection history (full disclosure):

1. V2 was chosen on development data only, before its official scores existed.
2. Nine fine-tuned candidates (three learning-rate and replay settings × three epochs) were scored on development data only. These were the held-out 20% of each benchmark's training split and held-out pretraining text. All nine failed the Wikipedia bits/byte guard, so the rule kept V2.
3. The rule's highest-scoring candidate, this model, was evaluated once officially, as registered.
4. A weight-interpolation phase was registered and run; its rule selected α = 0.5.
5. After seeing all of these official scores, we chose V2.1 as our submission.

What the scoring looks like. Three questions written for this demo, not taken from any evaluation set. Each answer is scored by its log-likelihood, as the benchmarks do. They illustrate the scoring; they are not evidence of accuracy.

Training data

Stage 1: pretraining data (identical to V2)

Human-written English only; no model-generated corpora.

Contamination controls.

• N-gram exclusion: any document sharing a 13-word n-gram with the train or evaluation splits of HellaSwag, ARC-Easy, PIQA or WinoGrande, or with the WikiText-103 validation and test splits, was removed.
• Domain blocks: all of wikihow.com and instructables.com (the sources of HellaSwag and PIQA) were blocked, removing 12,708 documents.
• Deduplication: exact, URL and MinHash near-duplicate removal across all sources.
• Limits: this is not a guarantee against paraphrased overlap.

Benchmark-targeted slice. A fastText classifier was trained to recognize text resembling benchmark training examples: 80% of the train splits of ARC-Easy/Challenge, PIQA, HellaSwag, WinoGrande, OpenBookQA, SciQ and CommonsenseQA, against length-matched web passages. The top 20% of pool documents by score form this slice. Benchmark items were never used as training text, and official evaluation splits were never used for selection or tuning.

Tokenizer. A 16,384-token byte-level BPE trained from scratch on 100M characters of FineWeb and English Wikipedia (training partitions only). A pre-registered comparison against new tokenizers trained on the V2 mixture kept this one.

Stage 2: fine-tuning data (new in V2.1)

• Format: each question is written exactly as lm-evaluation-harness v0.4.12 scores it, e.g. Question: …\nAnswer: <choice> for ARC and PIQA, and partial scoring for WinoGrande. Our formatter reproduces all 15,523 official evaluation requests exactly.
• Held-out 20%: chosen by a fixed hash, never trained on, and used only for model selection.
• Overlap removal: a training item was removed if its context matches an official evaluation item after normalization, or shares any 13-word sequence with one.

Training procedure

Stage 1: pretraining (identical to V2)

Held-out loss plateaued near 3.17 nats during the constant phase, then fell to 2.97 during the final decay.

Stage 2: benchmark-train fine-tuning (new in V2.1)

The answer score used in training is the quantity the harness's acc metric compares. A temperature of 10 in the candidate softmax leaves which answer ranks highest unchanged. Short pilot runs showed that without the temperature, or at a higher learning rate (6.25e-5), the loss damaged held-out text modelling.

Limitations and risks

• Task-tuned evaluation. The benchmark scores reflect training on those benchmarks' training splits. Do not use them as evidence of general reasoning ability or compare them directly with pretraining-only models.
• Small model, frequent errors. It produces fluent but often false statements. Do not use it for factual answers or advice.
• Not an assistant. There was no instruction, preference or safety tuning. The fine-tuning stage taught answer scoring, not instruction following, and there is no content filtering.
• General text modelling. It is slightly worse than V2 (WikiText-103 perplexity 25.46 vs 23.85). Use V2 for pure language-modelling research.
• Data biases. Web, encyclopedic, Q&A and older public-domain book text carry the social biases and dated viewpoints of their sources.
• English only, with a 1,024-token context.

Intended uses: research and teaching on small-scale pretraining and on what task-specific fine-tuning adds to a scratch-trained model; a baseline for other from-scratch models.

Environmental impact

• Stage 1 (V2): 31.9 H100-hours for the selected run, 35.8 H100-hours for all V2 GPU work.
• Stage 2: 20.5 minutes for V2.1 itself. The whole fine-tuning study (pilot runs, three arms, development selection, interpolation and official evaluations) came to under 1.5 H100-hours.
• Energy: power was not measured. At the H100 NVL's 400 W maximum board power, the total is at most about 15 kWh. Carbon emissions depend on the local grid and were not measured.

Licenses and attribution

The model weights are released under the MIT license declared above. The training data keeps its own terms.

Pretraining data:

• FineWeb-Edu: ODC-By 1.0.
• DCLM-baseline: CC-BY-4.0.
• Wikipedia: CC-BY-SA 3.0 / GFDL.
• StackExchange content: CC-BY-SA 4.0.
• Project Gutenberg: public domain.
• Common Crawl terms also apply to the web sources.

Fine-tuning data:

• ARC: CC-BY-SA 4.0, per its Hugging Face dataset card.
• WinoGrande: its GitHub repository (allenai/winogrande) declares Apache-2.0.
• PIQA: its Hugging Face dataset card lists the license as "Unknown".
• HellaSwag: its Hugging Face dataset card declares no license.
• For PIQA and HellaSwag, see the original authors' releases before redistributing derived data.

We thank the authors of these datasets and the maintainers of lm-evaluation-harness.

Files and reproducibility

• model.safetensors, config.json, generation_config.json, tokenizer.json: identical to the selected fine-tuning export (checkpoints/v2_1 in the repository).
• tokenizer_config.json: identical except tokenizer_class, which is set to PreTrainedTokenizerFast so that Transformers 4.x can load the tokenizer. Token IDs are unchanged (same file as the V2 release).
• figures/: the figures shown on this card, generated from the recorded results by src/scglm_v2/figures.py in the GitHub repository.
• training_provenance.json: the fine-tuning run's record (parent hash, data hashes, update and token counts) with V2's pretraining provenance nested inside.
• eval_results.json: the official metrics above and V2's as a reference, with the SHA256 of each original evaluation record.
• Full records (code, configurations, pre-registrations, frozen development-only selections, the adoption record and all evaluation outputs): https://github.com/davemaxuell/Global-Innovation-Build-Challenge-V2.

Citation

@misc{scglm_v21_2026,
  title  = {SCG-LM V2.1: a 46M-parameter language model trained from scratch, with benchmark-train fine-tuning},
  author = {davemaxuell},
  year   = {2026},
  url    = {https://huggingface.co/davemaxuellkr/Global-Innovation-Build-Challenge-V2}
}

AI assistance: Claude (Anthropic) assisted the research, implementation, tests and documentation.