Trida-7B-Preview

Introduction

🚀 Trida-7B-Preview: Block Diffusion Language Model

We introduce Trida-7B-Preview, a high-performance 7-billion parameter language model representing the first publicly released Block Diffusion Language Model to originate from Korea.

Model Overview

Architecture: Block Diffusion Language Model

Base Model: Continually pre-trained from the highly efficient Tri-7B model.

Korean Language Leadership Trida-7B-Preview sets a new benchmark for generative models in the region. To our knowledge, it is the:

• First Block Diffusion Language Model to be openly released in Korea.

• Best-performing diffusion language model in Korean among similar model sizes.

This model is a significant step forward for the Korean LLM community, demonstrating the effectiveness of the Block Diffusion paradigm for complex, multilingual tasks.

Key Highlights

• Block Diffusion Architecture: Trida-7B-Preview leverages the Block Diffusion architecture, combining the strengths of parallelized diffusion generation with autoregressive dependencies for improved efficiency, control, and flexible-length sequence generation.
• Multilingual Leadership: Specially optimized for Korean, English, and Japanese, offering robust performance across all three languages.
• Korean First: To our knowledge, Trida-7B-Preview is the first Block Diffusion Language Model to be openly released in Korea.
• Best-in-Class Korean Performance: It is the best-performing diffusion language model in Korean among models of similar size, setting a new benchmark for generative models in the region.

Model Specifications

Trida-7B-Preview

• Type: Block Diffusion Language Model
• Training Stage: Pre-training & Post-training
• Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm
• Number of Parameters: 7.76B
• Number of Layers: 32
• Number of Attention Heads: 32
• Context Length: 4,096
• Vocab Size: 128,256

🔄 Training and Methodology

We followed the methodology outlined in the Fast-dLLM-v2 approach (as seen in the model: Efficient-Large-Model/Fast_dLLM_v2_7B [https://huggingface.co/Efficient-Large-Model/Fast_dLLM_v2_7B]).

Continual Pre-training from Tri-7B: Trida-7B-Preview was continually pre-trained starting from our autoregressive model, trillionlabs/Tri-7B. This process was executed using a Block Diffusion training paradigm to transition the efficient base model into a highly capable generative model.

🚀 Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "trillionlabs/Trida-7B-Preview"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

prompt = "Hey Trida. Why don'y you try that?"
messages = [
    {"role": "system", "content": "You are Trida, created by TrillionLabs. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

# Fast-dLLM v2 style parallel decoding
gen_ids = model.generate(
    inputs["input_ids"],
    tokenizer=tokenizer,
    max_new_tokens=2048,
    small_block_size=8,
    threshold=0.9,
)

response = tokenizer.decode(
    gen_ids[0][inputs["input_ids"].shape[1]:], 
    skip_special_tokens=True
)
print(response)

You can also checkout our repo (https://github.com/trillion-labs/Fast-dLLM-Trida) for evaluation and demo.

Evaluation

We evaluated Trida-7B-Preview across a comprehensive suite of benchmarks assessing general reasoning, knowledge recall, coding abilities, mathematical reasoning, and instruction-following capabilities.

Benchmark Results

General Reasoning and Factuality

Coding

Mathematical Reasoning

Instruction Following

Limitations

• Language Support: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
• Knowledge Cutoff: The model's information is limited to data available up to Febuary, 2025.

License

This model is licensed under the Apache License 2.0.

Contact

For inquiries, please contact: info@trillionlabs.co