danish-lm-400m-sft-v33-avg-top3
SFT from jensjepsen/danish-lm-400m-base-ropext8048-v1, 3 epochs over
1,878,629 rows from 20 datasets. Root of this repo is the equal-weight average
of the three top checkpoints; each is also included as a subfolder with
optimizer state.
| revision | steps | epoch |
|---|---|---|
root (avg-top3) | mean of the three below | — |
step-25683 | 25,683 | 1.75 |
step-29352 | 29,352 | 2.00 |
step-40359 | 40,359 | 2.75 |
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("jensjepsen/danish-lm-400m-sft-v33-avg-top3")
t = AutoTokenizer.from_pretrained("jensjepsen/danish-lm-400m-sft-v33-avg-top3")
# single checkpoint:
m = AutoModelForCausalLM.from_pretrained("jensjepsen/danish-lm-400m-sft-v33-avg-top3", subfolder="step-25683")
Evaluation
Full test splits, chat-format, greedy. Run with scripts/eval_full_suite.sh.
| metric | avg-top3 |
|---|---|
| GSM8K (0-shot pass@1) | 18.38 |
| CitGen | 28.80 |
| SciQ (open-Q pass@1) | 14.90 |
| TextMan summary (chrF++) | 44.07 |
| TextMan rewrite (chrF++) | 48.45 |
| CitMC | 46.70 |
| SciQ-MC | 58.90 |
| ARC-Easy | 41.12 |
| ARC-Challenge | 27.30 |
| OpenBookQA | 35.40 |
PIQA (raw, length-norm logp) | 60.00 |
PIQA (chat-mc, letter generation) | 49.00 |
| GPQA-Diamond | 26.77 |
| SciQ (length-norm logp) | 42.80 |
| CitMC (length-norm logp) | 56.39 |
| ARC (length-norm logp) | 24.25 |
| IFEval-DA prompt-strict | 24.7 |
| IFEval-DA prompt-loose | 25.2 |
| IFEval-DA inst-strict | 38.9 |
| IFEval-DA inst-loose | 40.0 |
Both PIQA modes are listed because they differ by 11 points and the
conventions disagree: raw is the standard PIQA protocol (score each solution
as a continuation, pick the higher length-normalized log-prob), chat-mc asks
for a letter. Neither reproduces the 53.00 on the v31 cards, so the v31 figure
came from a third harness and is not comparable to either.
ICL schema/format induction
jensjepsen/danish-icl-schema-format-v3, exact match on the parsed answer.
| split | held out | exact | key-set |
|---|---|---|---|
eval_schema | schema | 56.7 | 93.0 |
eval_format | format | 85.3 | 96.4 |
eval_both | schema + format | 47.8 | 86.8 |
eval_format overlaps danish-ner-sft-v1, which trains brace_pair
(2,276 rows) and the [x]...[/x] delimiter pair via spans_bracket. Of its
three held-out formats only kv_eq (82.7) is unseen in this mix.
NER
jensjepsen/danish-ner-sft-v1, exact match on the parsed answer and micro-F1
over (type, span) pairs. Both splits are built from DANSK test text. FAITHFUL
is span-wrap only: the answer both strips back to the source passage and
carries at least one well-formed tag pair.
jensjepsen/danish-lm-400m-sft-v31-avg-top3 is the same mix without
danish-ner-sft-v1 or danish-icl-schema-format-v3; both models were scored
by the same script on the same day.
| split | metric | v31-avg-top3 | v33-avg-top3 |
|---|---|---|---|
eval | exact | 0.2 | 58.7 |
| entity-F1 | 0.7 | 70.0 | |
| span faithful | 0 | 73 | |
eval_format | exact | 0.0 | 40.1 |
| entity-F1 | 0.0 | 56.8 |
eval — formats seen in training, n=1500
| format | v31 exact | v33 exact | v33 F1 |
|---|---|---|---|
brace_pair | 0.0 | 69.9 | 77.6 |
tsv | 0.0 | 66.7 | 74.5 |
numbered | 0.0 | 64.2 | 72.3 |
json | 2.2 | 63.5 | 73.4 |
tagged | 0.0 | 63.4 | 72.2 |
kv_bracket | 0.0 | 58.9 | 71.7 |
kv_colon | 0.0 | 58.2 | 73.5 |
kv_arrow | 0.0 | 55.1 | 66.4 |
spans_angle | 0.0 | 52.3 | 62.5 |
spans_bracket | 0.0 | 50.0 | 65.3 |
spans_paren | 0.0 | 42.6 | 57.3 |
v33 span faithfulness on this split is 73% (strip 86, tagged 87, bare 12); v31 is 0% (strip 2, tagged 25, bare 2) — it emits tag-like output but does not reproduce the passage verbatim.
eval_format — formats held out of both danish-ner-sft-v1 and
danish-icl-schema-format-v3 training splits, n=1500
| format | v31 exact | v33 exact | v33 F1 | v33 faithful |
|---|---|---|---|---|
bracket_pair | 0.0 | 60.5 | 71.3 | — |
kv_eq | 0.0 | 53.1 | 66.4 | — |
spans_brace | 0.0 | 7.5 | 17.2 | 6% (strip 83, tagged 15, bare 70) |
Key-value formats transfer to unseen delimiters; span-wrap does not. Three
span-wrap delimiters are trained (spans_angle, spans_bracket,
spans_paren) and the fourth returns the passage untagged in 70% of rows.
| prompt mode | eval exact | eval_format exact |
|---|---|---|
icl | 60.4 | 42.9 |
both | 63.1 | 38.0 |
instruction | 48.4 | 36.0 |
Checkpoint comparison
Full test splits, scripts/eval_downstream_once.py, ICL capped at 1000 rows.
| model | GSM8K | CitGen | SciQ | IFEval inst-strict | ICL | mean |
|---|---|---|---|---|---|---|
step-25683 | 17.2 | 25.4 | 12.3 | 37.3 | 51.9 | 28.82 |
step-29352 | 15.3 | 26.8 | 13.0 | 36.6 | 52.8 | 28.90 |
step-40359 | 14.8 | 26.8 | 14.4 | 37.6 | 51.0 | 28.92 |
| avg-top3 | 18.5 | 29.7 | 15.5 | 36.8 | 55.3 | 31.16 |
Training
| base | jensjepsen/danish-lm-400m-base-ropext8048-v1 |
| tokenizer | jensjepsen/danish-tokenizer |
| rows | 1,878,629 train / 98,876 eval |
| epochs | 3 (44,031 steps) |
| batch | 128 x grad-accum 1, max_length 8048 |
| optimizer | adamw_torch_fused, fp32 master weights, bf16 autocast |
| LR | 3e-5, constant_with_warmup, 500 warmup |
| attention | FlashAttention-2, DataCollatorWithFlattening |
| kernels | torch.compile, Liger off |
| hardware | 1x H100, 5h01m |
| final train loss | 0.598 |
wandb: da_sft_v33_full_mix20_bs128_fa2_compile
Datasets
jensjepsen/danish-metamath-gsmjensjepsen/danish-algebra-sft-v5-mixedjensjepsen/danish-arith-chain-sft-v1jensjepsen/danish-wiki-grounded-sft-v3jensjepsen/danish-text-to-question-v2jensjepsen/danish-sciqjensjepsen/danish-gsm8kjensjepsen/danish-instruction-following-v4jensjepsen/danish-wiki-closedqa-v1jensjepsen/danish-word-problems-v2jensjepsen/danish-wiki-closedqa-stem-v1jensjepsen/danish-wiki-broadqa-stem-v1jensjepsen/danish-wiki-mc-letters-v1jensjepsen/danish-rc-v1jensjepsen/danish-reason-v1jensjepsen/danish-textman-v1jensjepsen/danish-arcjensjepsen/danish-openbookqajensjepsen/danish-ner-sft-v1jensjepsen/danish-icl-schema-format-v3