DeepSeek-V4-Flash-0731 120B REAM 104E Healed BF16

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.

This is a direct-saliency REAM expert-pruned/merged derivative of deepseek-ai/DeepSeek-V4-Flash-0731. It was produced directly from the original 256-routed-expert checkpoint; no intermediate REAP-pruned checkpoint was used.

The REAM checkpoint was subsequently healed with assistant-only SFT. All low-rank updates and native router updates are merged into this repository; no PEFT adapter or other runtime attachment is required.

Architecture

The source repository's DSpark/MTP speculative-decoding auxiliary weights are not included. The standard autoregressive causal model is preserved, and num_nextn_predict_layers is set to 0.

REAM procedure

Calibration used 1,024 sequences of 512 tokens (524,288 tokens total), sampled and converted to the DeepSeek-V4 wire format from the private Baekpica/K-EXAONE-236B-REAP-calibration-mix dataset, revision 94e90afc261634a30384d3037d9021f1c997b286.

• Direct expert saliency: mean routing-weighted expert-output L2 norm.
• Similarity: equal-weight average of router-logit geometry and accumulated routing-weighted expert-output similarity.
• Per layer, the 104 highest-saliency experts seed bounded pseudo-groups (group_size=16).
• Neurons are aligned using Hungarian matching over activation and weight features (activation PCA rank 64), then merged with saliency weights.
• The first three hash-routing layers have their token-to-expert tables remapped to the merged experts. Group collisions leave an average of about 5.48 unique experts across six route slots; repeated slots are accumulated with their routing weights by the DeepSeek-V4 MoE implementation.
• Merged hidden states are propagated layer by layer; all layer checkpoints, similarity matrices, and checksums are retained in the build artifacts.

Implementation references:

• Cerebras REAP overview
• REAM paper
• SamsungSAILMontreal/ream, revision 84a3030716a0059589e9d10e2ea049e32b76cfa6
• Baekpica/solar-ream, revision d0201d1f484a521b4ec13a2e30a3ecb5a19a85b8

Machine-readable build details are in ream_metadata.json; the final structural audit is in checkpoint_audit.json, and hash-table remapping statistics are in hash_router_audit.json. Per-layer direct-REAP distribution and route-coverage diagnostics are in saliency_audit.json.

Standalone healing

Healing used the private, audited Baekpica/DeepSeek-V4-Flash-0731-REAM-Healing-Mix-2048 dataset. The four organic configs (seq1024, seq4096, seq8192, and seq16384) are pairwise identity-disjoint. Two additional procedural configs, needle8192 and needle16384, place exactly checkable records at front, middle, and back positions in near-limit haystacks.

The merged adapter path contains 2,328 single-example optimization steps:

Rank-8, alpha-16 residuals were trained on 467 attention/shared-expert linear modules, while the 43 native router matrices were updated at a much lower learning rate. Routed 3-D expert tensors remained frozen. The final residuals were merged into BF16 weights and the router matrices were saved directly. healing_metadata.json records every phase report, dataset checksum, adapter checksum, learning-rate provenance, and runtime_adapter_required=false.

needle16384 (14,737–15,726 tokens) is published as prepared training data but was not used in this checkpoint; the final near-limit phase used only needle8192.

Loading

Transformers 5.14.1 or newer is recommended.

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.models.deepseek_v4.modeling_deepseek_v4 import (
    DeepseekV4PreTrainedModel,
)

model_id = "Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-BF16"
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Keep all learned Parameters in their stored BF16 dtype. The 40 non-Parameter
# router-correction buffers remain FP32 for routing stability.
DeepseekV4PreTrainedModel._keep_in_fp32_modules_strict = set()
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="bfloat16",
    device_map="auto",
)

DeepSeek-V4 does not use a conventional Jinja chat template. The source encoding/ utilities are included for prompt encoding and output parsing.

Limitations

Expert merging changes model behavior, and short healing does not restore all source behavior. A deterministic smoke prompt, 17 * 23, returned 289 instead of 391 after healing.

Two held-out eight-domain generation passes are included as final_generation_smoke_8cases.json (512-token cap) and final_generation_smoke_8cases_2048.json (2,048-token cap). After correcting an erroneous math answer key in the first checker, the 512-token pass fully satisfied 6/8 case rubrics (23/26 checks), while the 2,048-token pass satisfied 7/8 (24/26 checks). The Python LRU response improved from a truncated 512-token answer to a complete 927-token response with EOS and all four code checks. The 7,440-token held-out retrieval prompt returned the exact requested JSON and all three checks passed. Korean instruction formatting, structured JSON, English model-systems explanation, and the science response also completed with EOS.

The quantitative-reasoning case is a material limitation. Its correct capacity is 900 liters, but the model made an arithmetic error and settled on 600 liters, then continued a repetitive self-correction loop to the full 2,048-token cap without EOS (72.27% repeated 4-grams). The report preserves both the original checker outputs and the corrected quality_check_version provenance. These observations are diagnostics, not benchmark scores.

The procedural needle data and held-out generation checks do not establish general long-context quality. Reasoning delimiters, language selection, factuality, repetition, and task-specific accuracy should be evaluated on the intended workload. No claim is made that this checkpoint preserves the source model's published benchmark scores. The omitted DSpark/MTP auxiliary means it must be served autoregressively without the source model's MTP speculative configuration. The source model's MIT license and usage considerations apply.