BIJA-150M: 149M-Parameter English Causal Language Model

BIJA-150M is a 149-million-parameter English causal language model trained from scratch on approximately 5.84 billion tokens. This small Transformer language model is compatible with PyTorch and Hugging Face Transformers and is intended for text generation, language-model research, and educational experiments.

It is designed for research, education, experimentation, and studying the complete language-model training pipeline—from tokenization and dataset preparation to training, checkpointing, and Hugging Face deployment.

BIJA-150M is a base language model. It is not instruction-tuned and should not be expected to behave like a conversational assistant.

Model Summary

Architecture

BIJA-150M uses:

• 18 Transformer blocks
• Grouped-query attention with 12 query heads and 4 key/value heads
• Rotary positional embeddings
• RMSNorm
• SwiGLU feed-forward layers
• Tied input and output embeddings
• A 512-token context window

The model contains custom Transformers code in configuration_bija.py and modeling_bija.py.

Training

The model was trained from scratch using causal language modeling.

Training configuration:

• Training tokens: 5,839,978,496
• Global batch size: 32,768 tokens
• Optimizer: AdamW
• Learning rate: 3e-4 peak, decayed to 3e-5
• Warmup: 1% of training
• Adam betas: (0.9, 0.95)
• Weight decay: 0.1
• Gradient clipping: 1.0
• Mixed-precision training: FP16 autocast
• Gradient checkpointing: enabled
• TF32: enabled where supported

Although mixed-precision training was used, the released model weights are stored in FP32 for stable and portable inference.

Training Data

The code portion includes Python, JavaScript, TypeScript, Java, C, C++, and Rust. Stack-Edu examples were filtered to permissively licensed subsets before processing.

See NOTICE.md for dataset and upstream licensing information.

Tokenizer

BIJA-150M uses a custom byte-level BPE tokenizer with:

• Vocabulary size: 32,768
• pad_token_id: 0
• bos_token_id: 1
• eos_token_id: 2
• unk_token_id: 3
• doc_token_id: 4

The tokenizer and model must be used as a matched pair. Do not replace the tokenizer with a GPT-2, Llama, or other tokenizer.

The model supports a maximum context length of 512 tokens. Inputs longer than 512 tokens should be truncated before inference.

Documents in the training data use the format:

<|doc|> document text <|eos|>

Usage

Because BIJA-150M uses a custom architecture, loading requires trust_remote_code=True.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Betrayedchair24/BIJA_150M_BASE"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
).eval().to(device)

prompt = "<|doc|>The future of artificial intelligence is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.05,
        pad_token_id=0,
        eos_token_id=2,
        remove_invalid_values=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For deterministic generation, set do_sample=False.

Implementation Notes

The released repository includes a corrected RoPE-cache implementation for Hugging Face loading. This ensures that the rotary positional-embedding buffers are rebuilt correctly when the model is loaded through from_pretrained().

The model does not currently use a KV cache, so long generation sequences may be slower than production-oriented language models.

Evaluation

The reported training validation metrics are:

• Best validation loss: 2.8157
• Best validation perplexity: 16.70

These metrics are internal held-out validation results. No external benchmark claims are made for this release.

For reproducible evaluation, report the model revision, tokenizer hash, dataset split, prompt format, context length, decoding settings, number of examples, and hardware.

Intended Uses

BIJA-150M may be useful for language-model research, educational demonstrations, tokenizer and dataset experiments, small-scale text completion, custom Transformer studies, and reproducible training experiments.

Limitations

BIJA-150M is a small base model and may generate repetitive or factually incorrect text, hallucinate information, produce biased or offensive content, perform poorly outside its training distribution, fail to follow instructions reliably, or generate insecure or incorrect code.

It is not safety-aligned and should not be used for medical, legal, financial, safety-critical, or autonomous applications.

License

The model code and released weights are provided under the Apache-2.0 license. This license does not replace or supersede the individual licenses and terms of the datasets used during training.

Citation

@software{bija_150m_2026,
  title  = {BIJA-150M},
  author = Amitabh Dey,
  year   = {2026},
  url    = {https://huggingface.co/Betrayedchair24/BIJA_150M_BASE}
}