TinyBrain-100M Base

A 103M parameter English causal language model trained from scratch.

TinyBrain-100M Base is a small LLaMA-style causal language model trained from scratch on the exnivo/tinybrain-pretrain-corpus-2b dataset.

This is a base model, not an instruct/chat model. It is intended for language modeling experiments, continued pretraining, supervised fine-tuning, and small-model research.

For chat or instruction-following behavior, use the instruction-tuned version:

exnivo/tinybrain-100m-instruct

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Photosynthesis is the process by which plants"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

At a Glance

Model Details

Intended Use

TinyBrain-100M Base is intended for:

• small language model research
• causal language modeling experiments
• continued pretraining
• supervised fine-tuning
• instruction tuning
• tokenizer/model experiments
• educational small-model projects
• comparing base vs instruct behavior
• lightweight local model experiments

This model is best used as a base checkpoint for further training.

Not Intended For

This model is not intended to be used directly as a finished assistant.

Do not rely on the base model for:

• polished chat behavior
• instruction following
• safety-critical answers
• factual authority
• medical, legal, or financial advice
• live/current information
• advanced reasoning
• production use without evaluation

For assistant-style behavior, use exnivo/tinybrain-100m-instruct instead.

Training Data

TinyBrain-100M Base was trained on:

exnivo/tinybrain-pretrain-corpus-2b

The pretraining corpus is a mixed-source English dataset containing factual text, educational text, math reasoning data, code data, conversation-style data, and clean web text.

The pretraining corpus scan found:

The training run used an estimated ~2.1B training tokens. Token counts may differ depending on tokenizer, packing, filtering, and training pipeline details.

Dataset Mix

The pretraining corpus includes these broad categories:

Relationship to TinyBrain

TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.

Pipeline:

TinyBrain Pretrain Corpus 2B
        ↓
TinyBrain-100M Base
        ↓
TinyBrain Instruct 200K
        ↓
TinyBrain-100M Instruct

Evaluation

A quick WikiText-2 evaluation was run on the base model.

This is a lightweight evaluation, not a full benchmark suite. Results may vary depending on evaluation script, tokenizer settings, context length, and dataset preprocessing.

Base Model Behavior

TinyBrain-100M Base is a raw pretrained model. It can complete text, but it is not tuned to follow instructions.

Example base prompt:

Photosynthesis is the process by which plants

The base model may continue with partially useful text, but it can also repeat, drift, hallucinate, or produce broken completions. This is expected for a small base model and is one reason instruction tuning is needed.

For better chat behavior, use:

exnivo/tinybrain-100m-instruct

Recommended Generation Settings

For raw base-model text completion:

temperature = 0.8
top_p = 0.9
max_new_tokens = 80
repetition_penalty = 1.08

For more stable completions:

temperature = 0.5
top_p = 0.85
max_new_tokens = 80
repetition_penalty = 1.1

For deterministic testing:

do_sample = False
max_new_tokens = 80

Example: Text Completion

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Gravity is the force that"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example: Fine-Tuning Starting Point

TinyBrain-100M Base can be fine-tuned on the TinyBrain SFT dataset:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig

base_model = "exnivo/tinybrain-100m-base"
dataset_id = "exnivo/tinybrain-instruct-sft-200k"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)

ds = load_dataset(dataset_id, split="train")

def format_example(example):
    text = ""

    for message in example["messages"]:
        role = message["role"]
        content = message["content"].strip()

        if role == "user":
            text += f"User: {content}\n"
        elif role == "assistant":
            text += f"Assistant: {content}\n"

    return {"text": text.strip()}

ds = ds.map(format_example)

config = SFTConfig(
    output_dir="tinybrain-100m-instruct-sft",
    dataset_text_field="text",
    max_seq_length=512,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    num_train_epochs=1,
    logging_steps=20,
    save_steps=500,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=ds,
    args=config,
)

trainer.train()

Training

TinyBrain-100M Base was trained from scratch on the TinyBrain pretraining corpus.

Training details:

Strengths

TinyBrain-100M Base is useful because it is:

• small and lightweight
• trained from scratch
• easy to inspect
• easy to fine-tune
• based on an open TinyBrain data pipeline
• trained on a compact mixed-source corpus
• suitable for small-model experiments
• useful as a base checkpoint for SFT

Limitations

TinyBrain-100M Base has important limitations.

The model may:

• hallucinate facts
• produce broken or repetitive text
• fail at math
• fail at instruction following
• misunderstand prompts
• generate incomplete code
• produce outdated or incorrect information
• drift off-topic
• repeat web/data artifacts

This is expected for a small base model. It has not been tuned to reliably follow user instructions.

For chat and assistant behavior, use the instruction-tuned model instead.

Suggested Evaluation

Recommended checks:

• validation loss / perplexity
• text completion quality
• repetition behavior
• short factual completions
• simple math completions
• code completion sanity checks
• hallucination checks
• before/after SFT comparison
• downstream instruction-following after fine-tuning

Example base-model prompts:

Paris is the capital city of

The Netherlands is a country in

A cat is an animal that

One plus one equals

Photosynthesis is the process by which plants

Citation

If you use this model, you can cite it as:

@misc{tinybrain_100m_base,
  title = {TinyBrain-100M Base},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-base}}
}

Related Repositories

• Pretraining corpus: exnivo/tinybrain-pretrain-corpus-2b
• Base model: exnivo/tinybrain-100m-base
• SFT dataset: exnivo/tinybrain-instruct-sft-200k
• Instruct model: exnivo/tinybrain-100m-instruct

License

This model is released under the Apache 2.0 license.

The training dataset is mixed-source and currently listed under license: other. Users should review the upstream dataset licenses and source metadata before commercial use of models trained or fine-tuned from this checkpoint.

Disclaimer

TinyBrain-100M Base is an experimental small base language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.

Do not use it for high-stakes applications without additional training, filtering, evaluation, and safeguards.