Tiyat Alpha
Para leer esta ficha en español, haz clic aquí.
Tiyat is a 184.1M-parameter multilingual foundation language model trained from scratch with majority representation of Indigenous languages of Mexico. The training corpus includes most of Mexico's 68 national Indigenous languages and their approximately 360 variants. To the best of our knowledge, Tiyat is the first LLM developed specifically for this linguistic diversity.
Unlike an adaptation of a pretrained model centered on English or another high-resource language, Tiyat arch1 started from random weights. Its tokenizer, training mixture, and decoder-only Transformer were built as part of the Tachiwin research pipeline. Indigenous languages account for a target 70% of the training mixture rather than being included as a marginal addition.
Intended use and expected behavior
Tiyat arch1 is a base autoregressive language model for research on multilingual representations, tokenization, and language modeling with low-resource languages at the center of the training distribution. It is suitable for short-form text continuation, qualitative generation experiments, reproducible education, and study of a small LLM trained entirely from scratch.
The model continues a prompt by predicting the next token. It is not instruction-tuned or optimized for chat, translation, or factual question answering. Prompts containing several words generally provide better signals about the intended language, register, and topic than short or ambiguous inputs.
Generation quality varies across languages and variants according to the available training data. Outputs may mix languages, lose coherence, or become repetitive, particularly in longer generations. The model has not yet undergone comprehensive evaluation for every language and variant, so generated text should be reviewed by a knowledgeable speaker when linguistic accuracy matters.
Architecture
Tiyat arch1 is a custom Transformer decoder-only causal language model implemented with JAX and Flax/NNX. It is not a GPT-2, Llama, or other existing model variant.
| Property | Value |
|---|---|
| Parameters | ~184.1M |
| Transformer blocks | 12 |
| Embedding dimension | 768 |
| Attention heads | 24 |
| Feed-forward dimension | 3,072 |
| Maximum context | 918 tokens |
| Vocabulary | 64,000 tokens |
| Input/output embeddings | Separate (untied) |
| Dropout | 0.1 |
| Objective | Causal next-token prediction |
| Framework | JAX, Flax/NNX, Optax |
The architecture uses learned token and positional embeddings, pre-normalized causal multi-head attention, GELU feed-forward layers, residual connections, dropout, a final layer normalization, and an independent output projection.
This repository includes both the original NPZ training artifact and a Safetensors conversion. For a Safetensors-only distribution, see tachiwin/tiyat_alpha_arch1_safetensors. Both repositories represent the same arch1 model and architecture.
Training data
- Dataset:
tachiwin/tiyat-ground-pretrain-m1024 - Tokenizer:
tachiwin/tokenizer_64k - Approximate training volume: 300M tokens
The target training mixture was:
| Category | Target share |
|---|---|
| Indigenous languages of Mexico | 70% |
| Spanish | 7.5% |
| English | 7.5% |
| Code | 15% |
The corpus is multilingual and uneven: not every language or variant has the same volume, quality, domain coverage, or degree of review. Consult the dataset card for data-specific documentation and licensing conditions.
Training configuration
| Setting | Value |
|---|---|
| Batch size | 4 |
| Learning rate | 0.0003 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Warmup ratio | 0.1 |
| Training steps | 141,672 |
| Final training loss | 3.4996705055236816 |
The final loss is a training statistic, not a cross-language quality score. The model has fewer than two training tokens per parameter and is primarily data-limited.
Usage
Install the runtime dependencies:
pip install jax "flax>=0.10" safetensors transformers huggingface_hub
The repository ships custom configuration and modeling code. Review that code before enabling trust_remote_code=True.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "tachiwin/tiyat_alpha_arch1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
)
prompt = "Había una vez un gato que"
input_ids = tokenizer(prompt, return_tensors="np")["input_ids"][0].tolist()
output_ids = model.generate(
input_ids,
max_new_tokens=40,
temperature=0.7,
top_k=40,
top_p=0.9,
repetition_penalty=1.5,
no_repeat_ngram_size=3,
eos_token_id=tokenizer.eos_token_id,
seed=0,
)
print(tokenizer.decode(output_ids, skip_special_tokens=True))
model.generate() is a minimal implementation included with this repository. It supports greedy decoding or temperature sampling with top-k, top-p, repetition penalty, and no-repeat n-gram blocking. It is not the complete Transformers GenerationMixin: beam search and GenerationConfig are not currently supported.
Open research and technological sovereignty
Tiyat is part of Tachiwin's work toward technology that communities can study, reproduce, modify, and shape—not merely access as a closed external service. The project documents the broader pipeline, including document digitization, corpus construction, tokenization, architecture, training, and inference.
- Technical and project article
- Reproducible training and inference tutorial
- Public demo on Hugging Face
- Tachiwin
License
The model repository is released under the Apache License 2.0. Linked datasets, source documents, and other resources may have their own licenses or conditions; consult their respective repositories before reuse.
About Tachiwin
Tachiwin is an Indigenous, Mexican, community-led, nonprofit, open-source initiative developing AI and language technology for the Indigenous languages of Mexico. Its horizon is technology capable of identifying, translating, and conversing across the country's 68 national Indigenous languages and their variants, guided by technological sovereignty, collaboration, non-extractivism, and shared knowledge.