ADAPT-Llama3.1-4.8B-Instruct

Model Description

ADAPT is a technique that allows for size interpolation across different post-trained variants of the same base model. This is the student model distilled from Llama-3.1-8B-Instruct from our paper.

Training Procedure

This model was initialized from Llama-3.1-8B-Instruct by copying every other layer and the last 2 layers. It was distilled on 1B tokens of The deduplicated Pile and 1B of the math split from Llama Nemotron Post Training Dataset with cross entropy, KL, and cosine loss to match the activations of Llama-3.1-8B-Instruct. We used the following hyperparameters:

• Learning rate: 3e-4
• Learning rate scheduler: cosine
• Warmup ratio: 0.01
• Optimizer: AdamW
• Adam betas: (0.9, 0.95)
• Adam epsilon: 1e-8
• Weight decay: 0.1
• Max. gradient norm: 1.0
• Number of training steps (Pre-training phase): 480
• Effective batch size (Pre-training phase): 2048
• Number of training steps (SFT phase): 585
• Effective batch size (SFT phase): 4096
• Max. sequence length: 1024
• Mixed precision: bf16
• KLDiv weight: 0.1
• Cosine distance weight per layer: 10.0

Use

To interpolate between this model and Llama-3.1-8B-Instruct, please use the build_intermediate_model function from our github repository:

import torch
from patching.patch import build_intermediate_model

intermediate_model = build_intermediate_model(
  teacher_name_or_path = "meta-llama/Llama-3.1-8B-Instruct",
  student_name_or_path = "Harvard-DCML/ADAPT-Llama3.1-4.8B-Instruct",
  num_layers_to_patch = 2,
  patch_first_k_layers = True,
  dtype = torch.bfloat16,
)

Notes:

1. Changing num_layers_to_patch changes the size of the intermediate model by patching different numbers of student layers.
2. patch_first_k_layers should be set to True for this model for optimal interpolation performance.

Citation

@misc{zhou2026thinkingrightsizeamortized,
      title={Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs}, 
      author={Yan Zhou and Sara Kangaslahti and Jonathan Geuter and Nihal V. Nayak and Marco Fumero and Francesco Locatello and David Alvarez-Melis},
      year={2026},
      eprint={2608.22854},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.22854}, 
}