Aivora Financial LLM (137M, trained from scratch)
A DeepSeek-V3-style language model built and trained from scratch for finance: Mixture-of-Experts feed-forward layers, multi-head latent attention (MLA) with compressed KV, and multi-token prediction (MTP).
This is a learning/research artifact, not a production assistant. It writes fluent finance-flavoured text, and its facts are often wrong.
What it actually scores
| metric | value |
|---|---|
| validation loss | 4.4258 (random guessing at this vocabulary is 10.82) |
| training steps | 247,850 |
| tokens seen | 1,964,851,200 |
| 45-question finance quiz, model generating alone | 1-5 / 45 |
| 45-question finance quiz, full demo pipeline | 45 / 45 |
| 18 held-out questions (written after the fixes), automated scorer | 13 / 18 |
| the same 18, graded by hand | 18 / 18 (11 calculations exact, 7 definitions accurate) |
Read those three rows together. The 45/45 is flattering: those questions guided the fixes, and the glossary now phrases answers the way the answer key does. The 13/18 is unflattering: the scorer matches strings, so a correct paraphrase ("Depreciation spreads the cost of a tangible asset ... across its useful life" vs "spreading the cost of a tangible asset over its useful life") counts as wrong. The honest summary is the third row.
The pipeline number is what this Space actually scores: exact arithmetic from the calculator (including multi-step identities - ROE from assets and liabilities, P/E from net income and share count, margins from components), checked definitions from a curated glossary, and model generation only when it passes a degeneracy guard. Every answer is labelled with its source.
For comparison, on the same 45 questions, Qwen2.5-1.5B-Instruct scores 38/45 without any training on this data. A 101M model trained on 1.9B tokens cannot compete with models pretrained on trillions - that is the honest conclusion of this project, not a bug.
Is it undertrained?
No - it ran the full planned schedule and then stopped improving, which was measured rather than assumed:
| what was tried | result |
|---|---|
| 4 pretraining sessions to the planned 247,850 steps | val settled at ~4.43; the last 10,488 steps (the learning-rate anneal to 1e-5) moved it by nothing |
| instruction tuning at 2e-5 and at 5e-6 | both overfitted within 500 steps |
| distillation from a 1.5B teacher | answers became fluent and well-formed; the facts stayed wrong |
| Mixture-of-Experts routing check | balanced across all 8 layers (entropy 0.95-1.00), so no dead experts and no hidden capacity |
The limit is capacity: 101M parameters trained on ~2B tokens. Style can be taught to a model this size; knowledge cannot.
Architecture
| parameters | 136,892,248 |
| layers | 8 |
| embedding dim | 512 |
| attention heads | 8 |
| experts / active per token | 8 / 2 |
| KV LoRA rank (MLA) | 128 |
| context length | 1024 |
| tokenizer | GPT-2 BPE (50257 tokens) |
Training
Trained on a free Kaggle T4 x2 across several sessions, on a mix of general and financial text (FineWeb-Edu, Investopedia, SEC/EDGAR filings, finance Q&A and instruction sets). Continuation runs used peak LR 3e-5, cosine decay to 1e-5, 500-step warmup after each resume, gradient clipping 1.0 and fp16 loss scaling.
How a question is answered
- figures present -> deterministic calculator (including multi-step identities: ROE from assets and liabilities, P/E from net income and share count, EBIT from EBITDA, growth between periods);
- a glossary term appears -> curated definition;
- asked in other words -> retrieval over the same glossary (MiniLM embeddings, TF-IDF fallback), so "how is a company financed by borrowing rather than issuing shares?" reaches the leverage entry;
- otherwise -> the model, shown only if it passes a degeneracy guard.
Retrieval is tuned for precision: below its threshold it returns nothing and the question falls through, because a confidently wrong definition is worse than an answer labelled unreliable. Every answer states which of the four produced it.
Numbers are not generated
The bundled demo routes any question containing figures to a deterministic calculator (EBITDA margin, current ratio, ROE, P/E and others). A model this size cannot be trusted with arithmetic, so it is not asked to do it.
Usage
import json, torch, tiktoken
from safetensors.torch import load_file
from models import DeepSeekConfig, DeepSeekV3
meta = json.load(open("config.json"))
model = DeepSeekV3(DeepSeekConfig.from_dict(meta["model_config"]))
model.load_state_dict({k: v.float() for k, v in load_file("model.safetensors").items()})
model.eval()
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor(enc.encode_ordinary("Question: What is EBITDA?\nAnswer:")).unsqueeze(0)
print(enc.decode(model.generate(ids, 64, temperature=0.7, top_k=40)[0].tolist()))
Limitations
- Facts are frequently wrong; do not use for decisions.
- No instruction tuning in these weights (tuning it overfitted at two learning rates).
- English only, 1024-token context.