SmolLM2-135M Depth Up-Scaled to 40 Layers

This model is an experimental depth-upscaled version of HuggingFaceTB/SmolLM2-135M.

The original 30-layer LlamaForCausalLM architecture was expanded to 40 decoder layers using Depth Up-Scaling (DUS). The resulting model was then subjected to continued causal language-model pretraining on WikiText-2.

This repository contains the complete model rather than a LoRA adapter.

Model architecture

The 40-layer model was created from two overlapping slices of the original model:

• first slice: source layers 0–19,
• second slice: source layers 10–29.

The copied layers contain independent parameter tensors.

Training

The model was trained using continued causal language-model pretraining.

• Dataset: Salesforce/wikitext
• Dataset configuration: wikitext-2-raw-v1
• Training objective: causal language modeling
• Block size: 256 tokens
• Precision: FP16
• Optimizer: 8-bit AdamW
• Learning-rate scheduler: cosine
• Gradient checkpointing: enabled
• Early stopping: enabled
• Hardware: NVIDIA GeForce RTX 3050 Laptop GPU, 4 GB VRAM

The run configured for five epochs was stopped automatically at approximately epoch 1.37 because the validation loss stopped improving.

WikiText-2 results

Depth Up-Scaling initially increased loss. Continued pretraining subsequently adapted the new architecture and produced lower loss and perplexity on WikiText-2 than the original model.

External benchmark results

The models were evaluated using lm-evaluation-harness in the zero-shot setting.

Although the DUS model achieved better loss and perplexity on WikiText-2, it performed worse on all three external benchmarks. This suggests stronger adaptation to the continued-pretraining dataset accompanied by reduced generalization and possible catastrophic forgetting.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "spahbod/smollm2-dus-40l"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
).to("cuda")

prompt = "The history of artificial intelligence"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For CPU inference, remove .to("cuda") and load the model using torch_dtype=torch.float32.

Intended use

This model is intended for:

• educational experiments with model depth expansion,
• research into Depth Up-Scaling,
• continued-pretraining experiments,
• comparison of in-domain perplexity and external benchmark performance.

It is not an instruction-tuned or chat model.

Limitations

• The model was continued-pretrained on the relatively small WikiText-2 dataset.
• Improvements observed on WikiText-2 did not generalize to HellaSwag, LAMBADA or PIQA.
• The model may produce incorrect, biased or incoherent text.
• The model should not be used as a factual authority.
• The experiment was performed on a single consumer GPU and was not a large-scale training run.

Reproducibility

Training code is available at:

https://github.com/s25132/smollm2-dus

Acknowledgements

This model is derived from HuggingFaceTB/SmolLM2-135M, released under the Apache 2.0 license.

The experiment uses the WikiText dataset and the EleutherAI Language Model Evaluation Harness.