Ling → GLM repair with bilingual SFT

The default branch contains the selected full-size repair followed by 128 steps of English/Russian SFT. The complete repair, real training restarts, continued training, upload, download and independent reload were validated on a Lambda H100 SXM5 80 GB.

This is a directional repair experiment. Russian factual errors and failures to retain dialog context remain; the model is not fully recovered.

Checkpoints and evidence

• Default branch: selected SFT model, promoted at 11ad41087a67908665c0a7c7f91fffdcc9a64499. Later commits add reports.
• sft-context128 branch: the same continuation; exact HF reload checked at 7dd885be27860737644d752f99ffea1f6ea56838.
• repair-c256-before-sft tag: repair-only predecessor at 5c5217ef6c9b5ec269d65d0f9fe2c0ecdbe960ad.
• training/continued_training/sft-context128/: final optimizer/RNG state, metrics, TensorBoard events, generated answers, source provenance, and validation reports.
• training/prior_trials/: consolidated reports from the other three debug repositories. Rejected SFT logs are under training/continued_training/.
• training/complete_validation.json: final acceptance. The earlier failed acceptance is preserved; successful repair evaluations were reused with a fresh phase/service audit.

The original patient and real Ling donor retain their original HF revisions. New versions use commits and branches of this repository.

Measured results

The original mini-set has 110 generated answers per model, with automatic keys for 88. The table applies the same contextual contradiction filter to both sides.

Final group scores: EN questions 40/40, RU questions 27/30, EN dialogs 8/9, RU dialogs 6/9. New context dialogs: EN24/24, RU23/24. Their exact prompts were absent from the supplied repair/SFT training corpora. These are development probes, not an external benchmark or a claim about pretraining contamination.

Automatic scoring remains imperfect: it can accept a correct antonym followed by an unrelated one, or a substring in an imprecise Russian answer. Known failures include horse as the barking animal, forgotten favorite color, and incorrect contextual arithmetic. Legacy scores, rejection evidence, and qualitative observations are retained; 81/88 is not comprehensive human semantic accuracy.

KDA relative error fell from 0.109082 to 0.062277. Phase A block loss fell from 0.434423 to 0.347766 with mHC frozen. Geometry conversion preserved logits exactly. Phase C teacher KL fell from 0.401841 to 0.216651, and DSA loss from 0.318879 to 0.276803.

After SFT, teacher KL is 0.227675 and DSA loss 0.242032: some teacher alignment is traded for improved text loss and context behavior. Restoring only pre-phase-C mHC raises KL to 0.293672 with all other final weights fixed, showing a positive 0.065996 contribution from learned mHC.

External assistant loss fell from 1.216459 to 1.161270 EN and 1.112095 to 1.084241 RU. This uses a fixed 12 conversations per language, with 9,071 EN and 5,371 RU assistant tokens, not all 160 heldout records. Data comes from pinned UltraChat, T-Wix, a bounded local MERA sample, and 130 synthetic context exercises. Raw SFT records are not republished.

Loading on a sufficiently sized remote GPU

Validated versions: Torch 2.13.0+cu126, Transformers 5.16.1, FLA 0.5.2. Training peaked at approximately 55 GB reserved VRAM. An 8 GB workstation is insufficient; the helper checks memory before loading weights. All model work ran remotely.

Use the supplied helper to preserve learned FP32 mHC and normalization values in this mixed BF16/FP32 checkpoint. Plain automatic dtype loading can round small updates away.

from pathlib import Path
import importlib.util
import torch
from huggingface_hub import snapshot_download

path = snapshot_download(
    "imvladikon/ling-glm53-repair-debug-20260908-spread-c256",
    ignore_patterns=["training/**"],
)
spec = importlib.util.spec_from_file_location("repaired", Path(path) / "load_repaired.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
tokenizer, model = module.load_repaired(path, device="cuda:0")
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Какая столица Франции?"}],
    add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
)
inputs = {key: value.to("cuda:0") for key, value in inputs.items()}
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    output = model.generate(
        **inputs, max_new_tokens=64, do_sample=False, logits_to_keep=1
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Every downloaded root checkpoint/config/helper file was checked by SHA256. Four EN/RU probes reproduced identical generations and logits, with maximum difference 0. The final optimizer state also matched its HF object by SHA256. All saved floating-point weights passed a full NaN/Inf scan.

Continuing training

All 7,804,614,960 text parameters were trainable. A real 64-to-128 process restart restored 3,008 optimizer-state tensors and RNG. Small normalization weights remain FP32; weights are not quantized, while bitsandbytes AdamW uses 8-bit optimizer state.

Training uses the private pipeline's pinned VERL KDA backward and DSA auxiliary fixes. Unmodified stock Transformers training, Slime/Megatron/SGLang execution, and GRPO were not tested. All four fork revisions are recorded in the lock file.

The final sft_resume.pt matches this checkpoint and preserves the completed run's optimizer/RNG state. Additional training starts a new run from the published model with a new schedule; an existing completed run cannot resume with altered identity arguments.

Source, reproducible commands and fixes are in the owner's private GitHub pipeline.

W&B server histories/tables and TensorBoard events were verified: training, full/extended evaluation, new context evaluation.