ADAPT-Qwen3-2.3B-Base

Model Description

ADAPT is a technique that allows for size interpolation across different post-trained variants of the same base model. This is the student model distilled from Qwen3-4B-Base from our paper.

Training Procedure

This model was initialized from Qwen3-4B-Base by copying every other layer and the last 2 layers. It was distilled on 0.5B tokens of The deduplicated Pile and 0.5B of the math split from Llama Nemotron Post Training Dataset with cross entropy, KL, and cosine loss to match the activations of Qwen3-4B-Base. We used the following hyperparameters:

• Learning rate: 3e-4
• Learning rate scheduler: cosine
• Warmup ratio: 0.01
• Optimizer: AdamW
• Adam betas: (0.9, 0.95)
• Adam epsilon: 1e-8
• Weight decay: 0.1
• Max. gradient norm: 1.0
• Number of training steps (Pre-training phase): 240
• Effective batch size (Pre-training phase): 2048
• Number of training steps (SFT phase): 293
• Effective batch size (SFT phase): 4096
• Max. sequence length: 1024
• Mixed precision: bf16
• KLDiv weight: 0.1
• Cosine distance weight per layer: 10.0

Use

To interpolate between this model and a post-trained variant of Qwen3-4B-Base through weight-delta transfer, please use the build_intermediate_model function from our github repository:

import torch
from patching.patch import build_intermediate_model

intermediate_model = build_intermediate_model(
  teacher_name_or_path = "Qwen/Qwen3-4B-Instruct-2507", # Or other post-trained variants of Qwen3-4B-Base (e.g. Qwen3-4B-Thinking-2507, Qwen3-4B)
  student_name_or_path = "Harvard-DCML/ADAPT-Qwen3-2.3B-Base",
  weight_delta_student = True,
  untrained_student_base_name_or_path = None, # Path to student initialized from Qwen3-4B-Base
  untrained_student_finetuned_name_or_path = None, # Path to student initalized from the post-trianed variant of Qwen3-4B-Base
  num_layers_to_patch = 2,
  patch_first_k_layers = False,
  dtype = torch.bfloat16,
)

Notes:

1. Changing num_layers_to_patch changes the size of the intermediate model by patching different numbers of student layers.
2. patch_first_k_layers should be set to False for this model for optimal interpolation performance.
3. To initialize student model from the teacher, use train/save_untrained_student.py from our github repository.

Citation

@misc{zhou2026thinkingrightsizeamortized,
      title={Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs}, 
      author={Yan Zhou and Sara Kangaslahti and Jonathan Geuter and Nihal V. Nayak and Marco Fumero and Francesco Locatello and David Alvarez-Melis},
      year={2026},
      eprint={2608.22854},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.22854}, 
}