Tri-21B

Introduction

We introduce Tri-21B, our flagship large language model that redefines the efficiency frontier in LLM training. By achieving state-of-the-art performance with only 2.3T training tokens, we demonstrate that exceptional capabilities don't require excessive computational resources.

Key Highlights

• Unprecedented Training Efficiency: Trained on just 2.3T tokens—significantly less than comparable models—while achieving 70.3% average accuracy across MMLU/KMMLU/Global MMLU benchmarks
• Pushing the Pareto Frontier: With only 2.95E+23 FLOPs, Tri-21B outperforms models requiring 2-10x more compute, setting a new standard for efficient scaling
• Enhanced Reasoning: Modified training dataset mixture specifically optimized for reasoning capabilities
• Advanced Post-Training: Significantly improved RL training pipeline focusing on mathematical reasoning and everyday usage
• Multi-lingual: Specially optimized for Korean, English, and Japanese.

Our Tri-21B represents a paradigm shift in efficient model development. When comparing performance to training FLOPs, our model dramatically pushes the Pareto frontier—achieving performance comparable to or exceeding models like Qwen2.5-32B (74.6% at 3.46E+24 FLOPs) and Gemma 3 IT 27B (67.6% at 2.27E+24 FLOPs) while using approximately 8-12x fewer computational resources.

Model Specifications

Tri-21B

• Type: Causal Language Model
• Training Stage: Pre-training & Post-training
• Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm, and GQA
• Number of Parameters: 20.73B
• Number of Layers: 32
• Number of Attention Heads: 32 (Query) / 8 (Key, Value)
• Context Length: 8,192
• Number of Tokens Seen: 2.3T
• Vocab Size: 124,416

Training Efficiency Analysis

Our approach to training efficiency sets new benchmarks in the field. The following comparison demonstrates how Tri-21B achieves superior performance per FLOP compared to other state-of-the-art models of similar scale:

¹ Average of MMLU / KMMLU / Global MMLU (ja)
² Performance per FLOP relative to Tri-21B

This efficiency breakthrough enables organizations to deploy state-of-the-art language models without the traditional computational barriers, democratizing access to advanced AI capabilities.

Quickstart

Here is a code snippet with apply_chat_template that demonstrates how to load the tokenizer and model and generate text.

Tri-21B Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "trillionlabs/Tri-21B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Explain the concept of quantum computing in simple terms."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

vLLM & SGLang Deployment

vLLM and SGLang support for Trillion Model is on the way. Stay tuned!

Evaluation

We evaluated Tri-21B across a comprehensive suite of benchmarks assessing general reasoning, knowledge recall, coding abilities, mathematical reasoning, and instruction-following capabilities. We compare our model against state-of-the-art models of similar scale: Gemmma-3-IT-27B and Qwen3-32B to demonstrate its competitive performance.

• *Note that koIFEval, systemIFEval, and KoRuler are our in-house evaluation benchmarks adapted for Korean to better assess model capabilities in Korean language tasks.
• **Note that MT-Bench, KO-MT-Bench, and LogicKor use a 10-point scale.

Benchmark Results

Models compared:

• Tri-21B: Our flagship 21B parameter model
• Qwen3-32B: Qwen's 32B parameter model
• Gemma3-IT-27B: Google's Gemma 3 instruction-tuned 27B model

General Reasoning and Factuality

Coding

Mathematical Reasoning

Instruction Following and Chat

Base Model Evaluation

The following table shows the performance of Tri-21B base model (before instruction tuning) on key benchmarks:

Limitations

• Language Support: The models are optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
• Knowledge Cutoff: The model's information is limited to data available up to Febuary, 2025.

License

This model repository is licensed under the Trillion License.

Contact

For inquiries, please contact: info@trillionlabs.co