K2-Horizon-MoVA-36B-A4B-MLA

Experimental. This is not the standard IFM/K2-Horizon-MoVA-36B-A4B. Its attention is changed, and its output is different from the original model's output (see Quality). For normal work, use the original model or a quantization of the original. Use this model to try very long contexts (up to 512K tokens) on a computer with less memory.

K2-Horizon-MoVA-36B-A4B-MLA is an experiment based on IFM/K2-Horizon-MoVA-36B-A4B. Most of its attention layers use multi-head latent attention (MLA) instead of grouped-query attention (GQA). Thus its KV cache is 45% smaller than the original's in BF16, and 61% smaller with the default q8 cache. With these weights and the rq4 cache, 512K tokens of context need about 111 GB in total (a 192 GB Mac or larger; see KV cache size).

Weights: BF16 (no quantization). The original IFM weight files, plus one file with the MLA attention projections (model-mla-delta.safetensors).

How it is different from the original model

The original model uses grouped-query attention (GQA) in all 48 layers. Each layer keeps 8 key heads and 8 value heads of 128 values for each token: 2,048 values per layer.

This model is a hybrid:

• 38 layers use MLA. Each of these layers keeps one latent vector of 768 values and one RoPE key of 128 values for each token: 896 values per layer. The attention computes the keys and values of the 32 query heads from the latent.
• 10 layers keep GQA (layers 32, 35, 37, 39, 41, 42, 43, 45, 46 and 47). In these layers, MLA changes the output most. Thus these layers keep their original attention and their full cache.

The conversion follows TransMLA (arXiv:2502.07864), with calibration on text. Distillation from the original model then trained the MLA projections (healing). The routed experts, the value experts, the routers and the other weights are the original weights (in this repository's encoding).

KV cache size

• q8: the MLA latent and RoPE key in 8 bits. This is the default. Its effect on quality is very small (KL divergence +0.0004 against the BF16 cache).
• rq4: the MLA latent in 4 bits after a seeded Walsh-Hadamard rotation, the RoPE key in 8 bits. The model file selects rq4 when the context that you ask for does not fit as q8. Its effect on quality is small (KL divergence +0.004).
• The 10 GQA layers keep BF16 keys and values in all formats. They are 40 KiB of each token's cache.

Memory for BF16: 75.6 GB of weights. With 128K tokens in the q8 cache, about 88 GB in total. With 512K tokens in the rq4 cache, about 111 GB in total. (Totals include 2.5 GB for prompt processing.)

Longest context by unified memory, with macOS's default GPU memory limit (three quarters of memory). 512K tokens is the model's maximum context:

Quality

KL divergence from the original GQA model (BF16), on 31 held-out text windows of 1,024 tokens (31,713 positions). Lower is closer to the original model. All six versions use the same healed MLA weights:

Platforms

• Apple Silicon Macs with oMLX or mlx-lm (MLX). Enable Trust Remote Code: the model file k2_mla.py is in this repository.
• NanoSeedLM C engine (NanoSeedLM): Metal (Apple Silicon) and CUDA (NVIDIA GPUs with enough memory for 75.6 GB of weights). Hybrid MLA models need the engine version with hybrid support (pull request #14).

How to run

oMLX

1. Use an oMLX version with K2-Horizon support (mlx_lm.models.k2_horizon).
2. Download this repository to the oMLX model folder.
3. In the model settings, enable Trust Remote Code.

mlx-lm

from omlx.patches.k2_horizon import apply_k2_horizon_patch
apply_k2_horizon_patch()            # makes mlx_lm.models.k2_horizon available

from mlx_lm import load, generate
model, tokenizer = load("txgsync/K2-Horizon-MoVA-36B-A4B-MLA", trust_remote_code=True)

Long context

To use more context than q8 can hold, tell the model file the longest context that you will use. Do one of these:

• Set NSLM_MAX_CONTEXT=524288 in the environment of the process that loads the model.
• Set "mla_max_context": 524288 in config.json.

The model file then selects the cache format. To keep the BF16 cache, set "mla_kv": "bf16" in config.json. Read The KV cache in the NanoSeedLM documentation.

NanoSeedLM C engine

Build the engine from NanoSeedLM (the version with hybrid support, pull request #14). Then run:

nslm-chat --model /path/to/K2-Horizon-MoVA-36B-A4B-MLA "What is 17 * 23?"

nslm-serve gives an OpenAI-compatible server. Read the NanoSeedLM documentation for the options.

Limits

• This is an experiment. The model is not identical to the original model, and it is not a replacement for it. The KL divergence in the Quality section shows the difference.
• You must enable Trust Remote Code. The model file needs the K2-Horizon support of oMLX.
• Prompt processing is slow at long context. On an M4 Max, at 128K tokens of context, one 2,048-token prompt chunk takes about 45 s. Use prompt-prefix reuse (oMLX keeps prefixes in memory and on SSD) so that each turn processes only its new tokens.
• oMLX calculates memory for the original GQA cache (192 KiB per token). Thus oMLX can admit less context than this model can hold.
• In the NanoSeedLM engine, hybrid MLA models need the engine version with hybrid support. Earlier versions do not load them.

Files

• model-000NN-of-00048.safetensors: the original BF16 weights of IFM/K2-Horizon-MoVA-36B-A4B.
• model-mla-delta.safetensors: the MLA attention projections of the 38 MLA layers.
• k2_mla.py: the model file (MLX).
• config.json, mla_config.json: the configuration, with mla_ranks (768 for an MLA layer, 0 for a GQA layer).

Earlier versions: the tag pre-t03 keeps the all-MLA version (84 KiB of BF16 cache per token, KL divergence about 0.15). The tag naive-j768 keeps the conversion before healing.

Credits

Base model: IFM, K2-Horizon-MoVA-36B-A4B (Apache-2.0). Methods: TransMLA (Meng et al., arXiv:2502.07864). Code: NanoSeedLM.

Original model card

K2-Horizon-MoVA-36B-A4B

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token. We have released the checkpoints, along with the data and the training code.

K2-Horizon-MoVA-36B-A4B Highlights

• Frontier-class results at 4B active parameters. On agentic and reasoning benchmarks it outscores open weight dense (approximately 30B model size) and MoE models up to 15× its size; and also performs competitively against closed frontier models (see Benchmark Results).
• 512K context. Native 524,288-token context from the midtraining stages onward.
• Intermediate checkpoints. Intermediate checkpoints are released so capability changes can be studied across training rather than at a single checkpoint.
• Fully open. Training data/recipe and the training code are public.

Benchmark Results

Quickstart

Serving

vLLM, recipe at recipes.vllm.ai/IFM:

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

Use an exact branch name from the inventory with vLLM's --revision option. For example, --revision pretrain_1100000 selects the final checkpoint of Pretraining, at step 1,100,000.

SGLang recipe validated on 2× H200 in the SGLang K2 Horizon cookbook:

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

API Usage

[!Tip] Recommended settings: reasoning_effort="high", temperature=1.0, top_p=0.95. Reasoning depth is selected per request through chat_template_kwargs. Thinking is returned in reasoning_content and the answer in content.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="IFM/K2-Horizon-MoVA-36B-A4B",
    messages=[{"role": "user", "content": "Explain the result step by step."}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=32768,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "high", "tool_call_format": "xml"}},
)
message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

Our model supports multiple tool-call formats, which can be changed with chat_template_kwargs. The supported values are json, xml, and xml_typed. The default is xml. Keep --tool-call-parser k2_horizon enabled to parse the selected format.

Transformers

Validated with Transformers 5.15.0, PyTorch 2.13.0, Safetensors 0.8.0.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "IFM/K2-Horizon-MoVA-36B-A4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="bfloat16", low_cpu_mem_usage=True, trust_remote_code=True
)

inputs = tokenizer("Explain why long-context evaluation is difficult.", return_tensors="pt").to(model.device)
inputs.pop("token_type_ids", None)
outputs = model.generate(**inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Overview

The table below lists the training stages in order and the purpose of each stage.

Training steps are counted within each stage or phase. Token budgets cover only the additional training in that stage or phase. For example, the 50B tokens listed for SFT Phase 2 are additional to the 219B tokens in Phase 1, bringing the cumulative budget to 269B tokens by the end of Phase 2. Here, B and T denote billion and trillion tokens, respectively.

Each stage or phase continues from the final checkpoint of the preceding stage or phase.

Some stages, such as SFT, have multiple phases with slight changes to the data mix while retaining the same overall purpose.

Release Artifacts

The tables below list the release artifacts for K2-Horizon-MoVA-36B-A4B, their availability, and the expected release dates for remaining items.

Last updated: 2026-09-28

Status:

• Available — fully released for the scope listed;
• Partial — some items are available, with remaining items listed in the notes;
• In Progress — being prepared for release but not yet available.

Artifact Index

Checkpoint Inventory

Model repository: IFM/K2-Horizon-MoVA-36B-A4B

Branch names below refer to this repository. Patterns containing * group branches by training stage or phase. The * is a placeholder for a training-step number, not a literal branch name. Intermediate checkpoint groups exclude the final checkpoint listed separately; a pattern does not imply that a checkpoint is available at every step.

For example, sft_1_11000 is the checkpoint saved at training step 11,000 within SFT Phase 1, and is the final checkpoint of that phase. The numeric suffix is the step within the named stage or phase, not the cumulative step across all training. Thus, sft_2_2500 refers to step 2,500 within SFT Phase 2.

For a partially released group, the available checkpoints and the remaining checkpoints are listed in the notes.

[!IMPORTANT] Note:

• Released K2-Horizon Hugging Face checkpoints (e.g. Huggingface) can be used for inference, evaluation, and downstream fine-tuning (including SFT). Training behavior in the bundled Hugging Face implementation may differ from native xLLM, including the auxiliary load-balancing loss. To continue the original pretraining with xLLM's training behavior, use the native xLLM checkpoint and xLLM runtime.

Best Practices

1. Reasoning effort: always high. All reported results use high reasoning effort. Pass {"chat_template_kwargs": {"reasoning_effort": "high"}} on every request. medium and low effort settings are not recommended except for research on reasoning efforts.
2. Sampling parameters. temperature=1.0, top_p=0.95.
3. Serving. Use the validated SGLang recipe above: BF16, TP=2, FlashAttention-3, and the xllm_source_router_gemm_partitions override, which preserves the checkpoint's router numerics. Full recipes for every K2-Horizon size, with measured H200 latency and throughput, are in the SGLang cookbook and the vLLM recipe.
4. Parsers. Enable the k2_horizon reasoning parser for chat, and add the k2_horizon tool-call parser for agent use. Leave both off for plain completion-style generation.

Citation

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}