🐻 Gumini-1.5B (구미니)

5,700× less data, better performance.
Gumini-1.5B achieves Korean PPL 8.49 with only 3.14B tokens, outperforming Qwen-1.5B (18T tokens, PPL 8.84).

🔥 Key Results

📊 Data Efficiency

Model Description

Gumini-1.5B (구미니) is a bilingual Korean-English base language model trained using the Inheritune methodology. Starting from Qwen 2.5 3B, the model progressively grew from 10 to 16 layers through 7 training stages, with ~3.14B tokens of continued pretraining on a Korean–English mixed corpus.

This is a BASE model, not instruction-tuned.
It produces text continuations rather than conversational responses.

Training Highlights

Inheritune Progressive Layer Growing

Stage 0: 10 layers (1.08B) → 393M tokens
Stage 1: 11 layers (1.15B) → 393M tokens
Stage 2: 12 layers (1.23B) → 393M tokens
Stage 3: 13 layers (1.31B) → 393M tokens
Stage 4: 14 layers (1.39B) → 393M tokens
Stage 5: 15 layers (1.47B) → 393M tokens
Stage 6: 16 layers (1.54B) → 786M tokens ⭐
────────────────────────────────────────────
Total: 16 layers, 1.54B params, ~3.14B tokens

Model Details

Training Data

Total: 80% Korean, 20% English

Optimization

learning_rate: 2.0e-4
weight_decay: 0.1
lr_scheduler: cosine
warmup_ratio: 0.01
max_grad_norm: 1.0
precision: bf16
gradient_checkpointing: true
attention: PyTorch SDPA (Flash Attention)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "GuminiResearch/Gumini-1.5B-Base",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("GuminiResearch/Gumini-1.5B-Base")

prompt = "저는 구미니입니다."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    repetition_penalty=1.2,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using Pipeline

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="GuminiResearch/Gumini-1.5B-Base",
    torch_dtype="bfloat16",
    device_map="auto",
)

output = generator(
    "저는 구미니입니다.",
    max_new_tokens=100,
    temperature=0.7,
    repetition_penalty=1.2,
)
print(output[0]["generated_text"])

Evaluation

Model Family

Limitations

• Base model: No instruction-tuning or safety alignment
• High repetition risk: Use repetition_penalty >= 1.2
• May generate incorrect or outdated information
• Should not be used in sensitive or safety-critical contexts
• Knowledge cutoff based on training data

License

Qwen Research License (Non-Commercial)

This model is Built with Qwen and derived from Qwen 2.5 3B.

Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT.
Copyright (c) Alibaba Cloud. All Rights Reserved.

This model is for NON-COMMERCIAL / RESEARCH use only.
For commercial use, contact Alibaba Cloud.

References

Inheritune Paper

@inproceedings{Sanyal2024inheritune,
  title={Inheritune: Training Smaller Yet More Attentive Language Models},
  author={Sunny Sanyal and Ravid Shwartz-Ziv and Alexandros G. Dimakis and Sujay Sanghavi},
  year={2024},
  url={https://arxiv.org/abs/2404.08634}
}

Qwen 2.5

@misc{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  year={2024},
  url={https://qwenlm.github.io/blog/qwen2.5/}
}

Citation

@misc{gumini2025,
  title={Gumini-1.5B: Bilingual Korean-English Language Model via Inheritune},
  author={Gumin Kwon},
  year={2025},
  note={Built with Qwen. Trained with Inheritune progressive layer growing.},
  url={https://huggingface.co/GuminiResearch/Gumini-1.5B-Base}
}

Author

Gumin Kwon (권구민)

• LinkedIn: linkedin.com/in/devgumin
• HuggingFace: GuminiResearch