Skylar-980M-Cobol
🔬 Published as a transparent research for the Skylar framework, not as a general-purpose or SOTA model.
A 980M decoder-only COBOL specialist, trained 100% from scratch (random init, no warm-start, no external pretrained weights) and instruction-tuned to write, explain, modify and translate COBOL. Local, sovereign, and small enough to run on a modest GPU — built for legacy banking / insurance / PA settings where code cannot leave the building.
🔗 Watch it grow: https://skyl4r.ai
⚖️ Honest scope
- COBOL-only by design. It deliberately refuses other languages ("Python non rientra nelle mie competenze: sono specializzato esclusivamente in COBOL"). That focus is a feature, not a gap.
- Small & token-budget-limited (980M params, 20.37B pretrain tokens) → strong on COBOL syntax, generation and multi-turn edits, weak on multi-step arithmetic/abstract reasoning (a 0.9B capacity ceiling — see results). Expect a competent, honest assistant, not a reasoning engine.
- A research artifact of the Skylar from-scratch stack, not a finished product.
What it does well (real, measured)
- Writes correct, compilable COBOL from a request, and modifies it across turns (e.g. "add an
OVERFLOW check if the sum exceeds 9999" → correct
IF … DISPLAY 'OVERFLOW' ELSE …). - Explains COBOL constructs, knows its scope, and declines out-of-domain questions honestly instead of hallucinating.
- On the executable COBOL benchmark it beats every deployable 7B generalist (below).
Architecture
| Parameters | ~980M |
| Layers | 36 |
| Model dim (d_model) | 1536 |
| Attention heads | 12 (GQA, 4 KV heads) |
| Head dim | 128 (decoupled) |
| FFN | SwiGLU, d_ff 4096 |
| Normalization | RMSNorm (fp32) + QK-Norm |
| Positional | RoPE (θ=1e6) |
| Context window | 8192 (trained) · 16384 max (config) |
| Vocabulary | 48128 (code-aware BPE, digit-split) |
| Embeddings | tied (input/output) |
Training
- From scratch, random init — no warm-start, no third-party pretrained weights.
- Pretrain: 20.37B tokens, sequence length 8192.
- SFT: ChatML, assistant-only loss, on a curated COBOL instruction mix.
Results — head-to-head on COBOLEval
All models evaluated on the same harness, greedy decoding, seed 0, GnuCOBOL compile+execute,
official {NAME}.TXT scoring.
Ordered by pass@1. Skylar-980M-Cobol posts the highest compile-success rate of any model here (80.1% — above even the 14B COBOL-Coder), and is #1 on pass@1 among general-purpose, from-scratch models. The specialized COBOL-Coder (a 14B Qwen2.5-Coder fine-tune, not from-scratch) leads pass@1 — at ~14× the parameters and warm-started from a pretrained code model.
| Model | Params | CSR (compile rate) | pass@1 |
|---|---|---|---|
| Claude Opus 4.8 (reference ceiling, not deployable) | — | 96.6% | 81.5% |
| COBOL-Coder-14B (Qwen2.5-Coder fine-tune, not from-scratch) | 14B | 73.95% | 49.33% |
| Skylar-980M-Cobol (this model) | 980M | 80.1% | 7.5% |
| Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
| CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
| StarCoder2-7B | 7B | 48.6% | 0.0% |
→ At ~7× fewer parameters than the 7B baselines (and 14× fewer than COBOL-Coder), fully local and from-scratch, it posts the highest compile rate of any deployable model and out-performs the mainstream 7B code models on pass@1.
Conversational usability
Hand-evaluated on real multi-turn dev tasks (identity, generation, multi-turn modification, code comprehension, honest refusal): usable — concise, in-character, writes correct COBOL, keeps context across turns, and stays honestly in-scope.
Usage
Use the skylar library — no custom modeling code lives in
this repo; the architecture is provided by the package.
pip install skylar
import skylar
m = skylar.load("Skyl4r-Ai/Skylar-980M-Cobol")
print(m.generate(
"Scrivimi un programma COBOL che somma due numeri PIC 9(4) e stampa il risultato.",
system="Sei un esperto programmatore COBOL.",
max_new_tokens=400))
import skylar registers the nano-transformer architecture, so AutoModelForCausalLM works with
no trust_remote_code and no modeling files in the repo:
import skylar # registers the architecture
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
import torch
repo = "Skyl4r-Ai/Skylar-980M-Cobol"
model = AutoModelForCausalLM.from_pretrained(repo).eval()
tok = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))
# (wrap the turn in ChatML <|im_start|>…<|im_end|> as the skylar lib does)
Limitations
- Multi-step arithmetic / abstract reasoning is unreliable (0.9B ceiling) — verify generated logic.
- COBOL-only: it will refuse or re-frame non-COBOL requests.
- pass@1 is modest in absolute terms (COBOL is a hard, low-resource domain — even GPT-4o is at 16.4%); the value is being #1 among deployable models at a fraction of the size, fully local.
License & attribution
© A. Ivanovitch — Apache-2.0. Code & framework: github.com/2sophia/skylar