ALEA R2 MoE 1B-A215M (research snapshot)

Experimental research snapshot, not an official ALEA release. This checkpoint comes from an ongoing ALEA model effort. It is published so that others can inspect and test a small model trained only on lawful data. It is not instruction-tuned for general use, it has not been safety-tuned, and later ALEA models may differ in architecture, data and behaviour. Do not use it for legal advice.

R2 is a 1.05B-parameter mixture-of-experts language model with about 215M parameters active per token. It was trained from scratch for 30B tokens, mostly on U.S. legal, regulatory and financial text from the KL3M Data Project, plus synthetic and procedural data created by ALEA. It is strongest at continuing legal, regulatory and SEC-filing text and at short answers grounded in a passage you supply. It is weak at closed-book general knowledge, open-ended chat, code and arithmetic beyond one step.

Data provenance

This model is not Fairly Trained certified. It was, however, very carefully designed to train only on public-domain, CC0, permissively licensed or ALEA-created data. As one example: for encyclopedic world knowledge, only Wikidata (CC0) was used, not Wikipedia (CC-BY-SA). No web crawl and no copyrighted books were used.

This describes what the model was trained on. It is not a promise about outputs: any language model can produce common phrases or passages that resemble existing text.

Excluded on purpose: Wikipedia (CC-BY-SA) and every other CC-BY-SA source; web crawls; books under copyright; SHELF (held out as an evaluation set); every evaluation used below (decontaminated before training).

Versions used, for exact reproduction. The public datasets above are the sources; R2 trained on local builds of them, with these differences:

• OpenGloss-Edu: R2 used the text before the recurring fictional cast was renamed. The published opengloss-edu v1 renames 262 over-used character names, so it differs from R2's copy in those names.
• Procedural data: the decay used the proc-v1 build, which was later integrated, deduplicated and published as OpenLemma v0.3. R2 did not see openlemma-v0.3-pretrain as published.
• OpenGloss-Instruct: R2's decay used deterministic (non-LLM) documents only. The HF dataset marks the 594,627 documents of the main chat slice with in_r2_decay; the k-shot extraction and format round-trip families (1.5% of the decay) came from the same v1 build.
• OpenLemma v0.1: only the non-leaky families; five templated families and any document with fixed e-mail or memo headers were removed.
• OpenLemma v0.2 and OpenGloss-Facts were published after R2's data was frozen and are not in this model.

Quick start

pip install "transformers>=4.57" torch safetensors

Both the tokenizer and the model need trust_remote_code=True. The model code is plain PyTorch (no custom kernels) and runs on CPU or GPU. Load the weights in bf16; the router weights stay in fp32 automatically. Tested with transformers 4.57.6 and 5.19.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "alea-institute/alea-r2-moe-1b-a215m-snapshot"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16)
model.eval()

Device placement. The model loads on CPU. For a GPU, use model.to("cuda"), or pass device_map="cuda" to from_pretrained (that needs pip install accelerate). It fits on any GPU with 4 GB or more (bf16 weights 2.1 GB).

(a) Plain continuation: recommended for legal, regulatory and financial drafting

Write the start of the document and let the model continue it. tok(text) prepends the <|start|> token that began every training document.

prompt = "This Agreement shall be governed by and construed in accordance with"
inputs = tok(prompt, return_tensors="pt")          # ids start with <|start|> (id 0)
out = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Output (greedy, bf16):

the laws of the State of New York, without giving effect to the conflicts of
laws principles thereof.

         IN WITNESS WHEREOF, the parties hereto have caused this Agreement to be
duly executed and delivered as of the day and year first above written.

Declarative openings work much better than questions. "The elements of negligence are" works; "Question: What are the elements of negligence?\nAnswer:" often triggers a procedural drill instead (see Limitations).

(b) Chat with a context passage: recommended for grounded QA

The chat format is the OpenGloss-Instruct format. Put the source text in a <|context|> block, with each passage in a numbered <|snippet|>, and then ask the question in the same user turn. The model answers in one or two sentences, often quotes the source with <|quote|>…<|/quote|>, cites the snippet number with <|cite|>n<|/cite|>, and ends its turn with <|eot|>.

context = (
    "<|context|>\n"
    "<|snippet|>[1] Lease Agreement, Section 4 (Rent)\n"
    "Tenant shall pay Landlord a monthly base rent of $5,875.00, due on the first day of each calendar month. "
    "A late fee of five percent (5%) applies to any payment received after the fifth day of the month.<|/snippet|>\n"
    "<|snippet|>[2] Lease Agreement, Section 9 (Notices)\n"
    "All notices under this Lease shall be in writing and delivered by certified mail to the addresses "
    "set out in Schedule A.<|/snippet|>\n"
    "<|/context|>"
)
messages = [{"role": "user", "content": context + "\nWhat late fee applies to a late rent payment? Cite the source."}]
enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=False))

Output (greedy, bf16):

See source 1: <|quote|>A late fee of five percent (5%) applies to any payment received after the fifth day of the month.<|/quote|> <|cite|>1<|/cite|><|eot|>

The template renders exactly the training layout:

<|start|><|user|>
<|context|>
<|snippet|>[1] Title
passage text<|/snippet|>
<|snippet|>[2] Title
passage text<|/snippet|>
<|/context|>
question<|eot|>
<|assistant|>

An optional system turn (<|system|>\n…<|eot|>\n) is supported but was in only 15% of training conversations; in our tests it did not help and sometimes hurt.

Special tokens and generation

generation_config.json stops on <|end|> or <|eot|> (eos_token_id = [1, 7]). Other reserved control tokens in the vocabulary (ids up to 255) were never trained and should not be used.

Speed. The code is plain PyTorch. At load time it stacks the experts into batched tensors, and on the GPU the expert computation avoids host-device syncs. Greedy decoding, bf16:

Prefill of 1,500 tokens takes 0.05 s on the GPU. For the lowest latency on CUDA, use the static cache. transformers then compiles the decoding step with CUDA graphs; the first call takes about 10–40 s to compile, and it compiles again for each new batch size or cache length:

model = model.to("cuda")
out = model.generate(**inputs, max_new_tokens=256, cache_implementation="static")

On CPU:

• Generate in bf16. Decoding is limited by memory bandwidth, and fp32 is about half as fast.
• For scoring or long prompts, use fp32. Prefill is compute-bound, and bf16 is about 3x slower on AVX2 CPUs.
• Use one thread per performance core: torch.set_num_threads(6) and, on a 6P+4E Intel, taskset -c 0,2,4,6,8,10. Threads on efficiency cores or hyper-threads slow decoding (all 16 CPUs: 18 tok/s).
• transformers compiles the static cache only on accelerators. To compile on CPU, opt in through CompileConfig and have a C++ compiler installed.

Inference options (alea_moe_impl, default "auto", plus flags that switch each speedup off) can be passed to from_pretrained, e.g. alea_moe_impl="loop". They are documented in configuration_alea_moe.py.

Faster code, same weights (updated 2026-10). The first upload (revision e197d44) ran the experts as a Python loop: about 40 tok/s at batch 1 on the GPU. The current modeling code computes the same function:

• in fp32 it matches the old code to within 5e-5 in the logits, and 22/22 greedy generations are identical;
• in bf16 both versions are within the bf16 rounding noise of the fp32 model (top-1 agreement 98%, Δloss < 1e-4 nats/token);
• knowledge probe 89.0 (bf16) / 89.1 (fp32), with the same 662/662 predictions as before.

To reproduce the exact earlier outputs, load revision="e197d44".

Other runtimes. There is no vLLM, llama.cpp or TGI support yet. Weight names follow the Qwen2-MoE convention (mlp.gate, mlp.experts.{e}.{gate,up,down}_proj, mlp.shared_expert.*) to ease a port.

Model details

No stock transformers class matches this combination (full-vector QK-norm, pre-norm blocks, ungated shared expert), so the repository ships a custom implementation (modeling_alea_moe.py, plain PyTorch). Training-only parts (load-balancing loss 0.01, router z-loss 0.001, intra-document attention masking) are not implemented; the code is for inference.

Conversion check. The HF port was compared with the original OLMo-core checkpoint in fp32: maximum absolute logit difference 2.7e-5, top-1 agreement 100%, greedy generations identical on 21/21 prompts, and held-out loss equal to within 2e-8 nats/token. It also passed clean-environment tests on transformers 5.19 and 4.57 (loading, tied embeddings, fp32 router in a bf16 load, chat template ids equal to the training encoding, batched = unbatched generation, pipeline). Storing the weights in bf16 costs about 1.5e-4 nats/token (0.01%) relative to fp32.

Training

Data mix (share of tokens by bucket)

Statute text was a deliberate memorization target: the U.S. Code and eCFR were seen about 7 times each (the U.S. Code about 12 times counting the triples renders). Most other tier-3 sources (case law, filings, patents, .gov) were seen at most once, and the stable and decay documents are disjoint.

The decay mix (decay.v4) changed four things relative to the stable phase:

1. Chat data. It introduced OpenGloss-Instruct, a deterministic, verified chat-format set (grounded QA over KL3M, OpenGloss and Wikidata passages, abstention when the passage lacks the answer, snippet citation, quotation, extraction to JSON/YAML, and format conversion). The control tokens appear only in this 5.5% slice.
2. Procedural data. It replaced part of the repeated OpenLemma with proc-v1: code-verified pattern induction, logic, arithmetic, formats and small code-tracing exercises (later published as OpenLemma v0.3).
3. Removal of the word-sense format. All 43,144 OpenLemma documents in the bracketed-word sense format ("Which meaning of the word in brackets is used? … 1. … 2. …") were dropped and replaced with other OpenLemma documents, because the format was leaking into legal generations.
4. Cleaning of corrupted regulatory text. Documents with OCR/encoding damage (Commission?s, 47 C.F.R. ?, ofthe) were removed from the .gov and statutes/regulations buckets and refilled with clean documents. About 67% of the decay's FCC tokens had been corrupted.

All evaluation sets used below were removed from the training data by fail-closed decontamination before training.

Evaluation

All numbers are for this checkpoint (step 28,610) unless marked. "R2 end of stable" is step 23,746, before the decay. D3-final is ALEA's previous research model (222M parameters, dense, trained on the same family of data). The reference models are base models run with the same harnesses.

1. ALEA probes

These are ALEA's own capability probes, run automatically on every checkpoint. Scores are percentages; "–" means not run for that model.

• Knowledge probe: 662 four-way definition items (164 rare U.S. legal terms, 300 OpenGloss word senses whose headword was in training, 198 held-out headwords), scored by PMI; chance 25%. R2-final: legal 94.5, seen words 91.7, held-out words 81.3. The items are paraphrased so that no answer shares a 4-word span with training text, but the probe is built from ALEA's own domains and so favours ALEA models.

• ICL v2: 30 few-shot tasks × 60 items (binding, induction, logic, arithmetic, legal), k=4 greedy exact match, macro over 29 tasks (95% CI about ±1.9). Read the split, not the total. We sorted the tasks by whether the procedural decay data trains the same task family directly (e.g. constant addition, syllogisms, key-value recall, date deadlines). The decay raised the 18 proc-analog tasks by +14.9 points and the 11 novel tasks by +2.3, so about 90% of the decay gain is on families the decay data trains. On novel tasks R2 is about 21 points behind SmolLM2-135M.

• Closed-book facts: 2,032 Wikidata-derived items asked with held-out phrasings; normalized exact match on the first 12 words; "forward" asks for the value, "reverse" for the subject. The MC column ranks the gold value against 3 same-relation distractors by log-likelihood (chance 25%). R2 recognizes facts nearly as well as SmolLM2-135M but recalls about two thirds as many. In chat format (k=0) it gets 11.1% and never abstains on the 205 questions about obscure entities that are held out of the fact set (0%); it guesses.

• Chat probe (64 held-out items of the trained chat families; only ALEA's chat format can run it):

  Family	R2-final, chat format	R2-final, same items as plain text	R2 end of stable, chat format
  Grounded QA (20): token F1 / answer grounded in passage	55.9 / 95	31.2 / 80	9.6 / 35
  Multi-snippet QA (12): F1 / correct snippet cited	59.1 / 91.7	0 / 0	7.6 / 0
  Abstains when the passage lacks the answer (12)	100	0	0
  Extraction to JSON (10): parses / field accuracy	100 / 100	60 / 50	0 / 0
  Section citation (6) / definition quote (4)	66.7 / 100	0 / 0	0 / 0

  These items use new seeds but the same task families and phrasing styles as the training data, so they measure how well the format was learned, not generalization. On 169 new, hand-written grounded tasks (Section 3) the same skills score much lower: direct lookup 42%, extraction field accuracy 31%, abstention on absent answers 25%.

2. Likelihood benchmarks (KL3M-paper suite)

The likelihood-scored suite of the KL3M paper (lm-evaluation-harness 0.4.13), run on this checkpoint on 2026-10-07. Zero-shot unless marked. LegalBench = balanced accuracy, macro over 19 core tasks; CaseHOLD = acc_norm; LEDGAR and FPB (Financial PhraseBank, all-agree) = balanced accuracy; PMI = argmax of ll(answer | question) − ll(answer | start token). The general mean is over ARC-e, ARC-c, PIQA, HellaSwag (first 1,000), Winogrande, LAMBADA, SciQ, BoolQ, OpenBookQA and MMLU (100 per subject). Typical standard errors: LegalBench ±0.5, CaseHOLD ±1.4, LEDGAR ±1.2, FPB ±1.1, ARC/HellaSwag ±1.0–1.6; differences smaller than about twice that are ties.

• Legal: LegalBench (52.2) is level with SmolLM2-360M (53.2) and Qwen2.5-0.5B (51.9), and above KL3M-003-1.7B (49.9) and every model under 200M parameters. Most small models, R2 included, are close to chance on CaseHOLD (20% for 5 options) and LEDGAR. Financial sentiment (FPB) is weak and gets worse with 5 shots.
• General: a mean of 44.1 is above GPT-2, Pythia-160M and the earlier ALEA models, level with Pleias-350m (the other model trained only on open data), and about 5 points below SmolLM2-135M, which saw about 67× more tokens. MMLU is at chance for every model under 0.5B except Qwen2.5.
• Settings. R2 is scored at its 2,048-token context with <|start|> prepended to every context, in OLMo-core with bf16 autocast. The references were scored at their native context (up to 8,192 tokens) with lm-eval's default BOS handling. The PMI columns are the most sensitive to this difference.

3. Domain tests: strengths and weaknesses

Four hand-checked generation tests (about 800 prompts, greedy plus samples) compared R2 with SmolLM2-135M/360M, Qwen2.5-0.5B, KL3M-003-1.7B and, for chat, SmolLM2-360M-Instruct and Qwen2.5-0.5B-Instruct. An LLM judge (gpt-6-luna, 0–3 rubric, spot-checked by hand) scored drafting quality. Excerpts below are verbatim and greedy unless noted.

Strengths

• Supreme Court citations: 21/22 exact for This principle is well established. See <case>, (Qwen2.5-0.5B 10/22, KL3M-003-1.7B 5/22), with correct parallel cites in 35 of 37 cases. Gideon v. Wainwright → 372 U.S. 335, 83 S.Ct. 792, 9 L.Ed.2d 799 (1963).
• Contract drafting by continuation at or above KL3M-003-1.7B (judge mean 2.52/3 and 90% usable, vs 2.40 / 87% for KL3M-1.7B, 1.43 for Qwen2.5-0.5B and ≤ 1.00 for SmolLM2). Indemnification: the Buyer and its Affiliates and their respective directors, officers, employees, agents, representatives, successors and assigns (each, an "Indemnified Party") from and against any and all claims, losses, damages, liabilities, costs and expenses (including reasonable attorneys' fees and expenses) …
• Statute recall: Rule 10b-5 (a)–(c) and 42 U.S.C. § 1983 verbatim in 3/3 samples; 18 U.S.C. § 1001(a) 2/3. Doctrine stated as a memorized formula is usually right, e.g. the preliminary-injunction factors: (1) he has a substantial likelihood of success on the merits; (2) he is likely to suffer irreparable harm in the absence of preliminary relief; (3) the balance of equities tips in his favor; and (4) an injunction is in the public interest.
• SEC and Federal Register boilerplate: forward-looking-statement safe harbors, critical accounting estimates, Item 9A controls language and FR preambles (AGENCY: Federal Aviation Administration (FAA), DOT. ACTION: Notice of proposed rulemaking (NPRM). SUMMARY: The FAA proposes to adopt a new airworthiness directive (AD) for all Airbus SAS Model A319, A320, and A321 series airplanes …).
• Snippet citation: with 3–4 snippets, R2 cites the right snippet first in 83% of items; the reference instruct models never cite by number. When did the Halden Valley line close completely? Say which document. → The relevant passage is 3. It closed completely in 1969. <|cite|>3<|/cite|>. Grounded legal QA in chat was judged 2.60/3 (95% usable) vs 2.24 for SmolLM2-360M-Instruct.
• Format conversion (CSV/JSON/YAML round trip) in chat: 4/5 exact, keeping "0042" and "1.0" as strings.
• Low looping: 3% of generations loop, against 15–20% for SmolLM2 base models. Chat turns end with <|eot|> in 95–97% of cases.
• Argument validity in natural prose: 9/10 (references 3–7/10).

Weaknesses

• Fabricated lower-court authority. Supreme Court cites are reliable, but circuit and state cites in opinion prose are often invented: Baker v. Baker, 596 So.2d 924 (Fla.1992); Brooks v. Brooks, 596 So.2d 924 (Fla.1992) (one cite, two names). Real cases are sometimes attached to the wrong proposition. West v. Atkins is still wrong (369 U.S. 560 … (1962)).
• Wrong legal content in fluent prose. A witness who says, "I heard a crash," is offering hearsay (in-court testimony is not hearsay); promissory estoppel described as judicial estoppel.
• Wrong percentages. Dollar changes in filing text are right (6/6) but the percentages are wrong 6/6 ($9.0 million, or 15.0 percent where the truth is 20.0%), usually with an invented cause.
• Generic Question:/Answer: cues summon procedural drills. Question: What is the business judgment rule?\nAnswer: → trinqvbhf\n\nQuestion:\nAre ((P ∨ P) ∨ (Q ∨ P)) ∨ … (a ROT13 drill). Legal Q/A keyword accuracy in this form is 1/14, vs 7/14 for SmolLM2-360M. Use declarative prompts or the chat format.
• Closed-book recall gaps. R2 recognizes many facts (likelihood) but produces few of them: 12/30 everyday facts in chat, 17/30 as cloze, and no fact correct in every prompt form (SmolLM2-135M: 15/30). Literature and people are nearly absent: Pride and Prejudice → James Joyce. It almost never abstains on nonexistent entities (2/18).
• No code. 0/32 on function bodies, traces, explanations, bug spotting and SQL; code prompts turn into procedural exercises (def is_even(n) → a function-equivalence drill).
• No summarization or open-ended writing in chat. Summarize the filing in one sentence. → The provision doesn't mention "Board of Directors", so I can't quote a definition from it. 0/10 summaries pass; chat drafting requests invent terms or quote a nonexistent provision.
• Extraction values. JSON/YAML output parses (94%) and keeps the requested keys, but only 31% of field values are right (Qwen2.5-0.5B-Instruct 68%), nested schemas are flattened, and absent fields are filled in.
• Invented quotes and answers by copying. About 9% of grounded chat answers contain a <|quote|> that is not in the passage, sometimes memorized EDGAR boilerplate ("Administrative Agent" means First Hawaiian Bank, N.A. … when the passage names a different bank). Answers copy the nearest sentence and never compute (0/21 on questions that need arithmetic over passage values).
• Word-level in-context learning. Antonyms 0/20 at k=4 (SmolLM2-135M 20/20); on 7 novel few-shot word tasks R2 scores 31% vs 63% for SmolLM2-135M.

Intended use

• Research on small models trained only on lawful data, including data-provenance, memorization and MoE studies.
• Continuation of U.S. legal, regulatory and financial text (clauses, statutory and regulatory language, filing boilerplate, opinion prose), with human review.
• Prototypes of grounded QA over passages you supply, in the chat format above.

Out of scope: legal advice or any use where output is relied on without verification; unsupervised factual QA (closed-book); code generation; general-purpose assistant chat; non-English text.

Limitations and risks

• Not instruction-tuned for general use. The chat format was learned from a 5.5% slice of the decay and covers grounded lookup, citation, abstention and extraction only. Open-ended requests (write, summarize, explain, draft in chat) are mapped onto those families.
• Hallucinated legal authority. Always verify citations, quotations and numbers. Lower-court citations, pin cites and percentages are the least reliable. Real names (people, companies, banks) can appear in invented documents.
• Not legal advice. Output is not reviewed by a lawyer and may be wrong on the law in fluent, confident prose.
• 2,048-token context. Longer inputs are untested.
• English-centric and U.S.-legal-centric. The EU Official Journal (English) is the only non-U.S. legal source.
• Register. Everyday plain prompts often get a short-sentence children's-reader register from OpenGloss-Edu (Why is the sky blue? The sky looks blue because → Earth spins.\n\nEarth spins like a top.).
• Templated outputs on Q/A cues. Question:/Answer: and some x -> y prompts trigger OpenLemma and procedural templates (logic formulas, string drills, **Header** sections).
• Phantom-passage abstention. Without a <|context|> block, chat answers sometimes refer to a passage that does not exist (Nowhere in the excerpt is "res judicata" mentioned …).
• No safety tuning. The model has not been red-teamed or aligned. It inherits the content of public legal and government records, which include descriptions of crimes, litigation and personal details in court records.

Citation

This is a snapshot from an ongoing ALEA model effort; please cite it as a snapshot.

@misc{alea_r2_moe_snapshot_2026,
  title        = {{ALEA} {R2} {MoE} 1B-A215M: a research snapshot of a small mixture-of-experts model trained on lawful data},
  author       = {{ALEA Institute} and Bommarito, Michael J., II},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/alea-institute/alea-r2-moe-1b-a215m-snapshot},
  note         = {Experimental research snapshot (step 28,610), not an official ALEA release}
}