Tinizong-50M

Tinizong-50M is a small experimental decoder-only language model.

The model was created primarily as a research and engineering platform for exploring language-model architecture, tokenizer design, dataset construction, pretraining, scaling, and instruction tuning.

This release contains a Hugging Face-compatible conversion of the native Tinizong checkpoint.

It is a base language model, not an instruction-tuned or chat model.

Model details

The native implementation was written directly in PyTorch and later converted to the Hugging Face LlamaForCausalLM architecture.

Training

Tinizong-50M was trained as a causal language model using next-token prediction.

Training data included a mixture of:

• Wikipedia-derived text
• synthetic factual / educational text
• other experimental pretraining material used during development

The training corpus and methodology evolved during the project, so this release should be considered an experimental research checkpoint rather than a fully documented production model. The model uses a custom 16K SentencePiece tokenizer with byte fallback.

Evaluation

Tinizong-46M was evaluated with the EleutherAI LM Evaluation Harness using the public Hugging Face checkpoint:

superAVTR/tinizong-46M

Evaluation settings:

• zero-shot
• Hugging Face backend
• batch size: 8
• model dtype: float32
• no custom evaluation code

For the multiple-choice benchmarks, acc_norm is the more useful comparison metric because it normalizes likelihood by completion length.

These results should be interpreted in the context of the model's small size (~46M parameters). The model performs above random-choice baseline on ARC-Easy, while HellaSwag and PIQA remain close to their respective random baselines. LAMBADA remains particularly challenging for this checkpoint.

The evaluation was run with:

#!pip install lm-eval

!lm_eval \
  --model hf \
  --model_args pretrained=superAVTR/tinizong-46M,dtype=float \
  --tasks lambada_openai,hellaswag,piqa,arc_easy \
  --device cuda:0 \
  --batch_size 8

Hugging Face conversion

The original Tinizong model uses a compact PyTorch implementation with:

• combined Q/K/V projection
• RMSNorm
• rotary position embeddings
• SwiGLU feed-forward layers
• tied input/output embeddings

The trained weights were mapped into Hugging Face's LlamaForCausalLM representation. The combined native QKV projection was split into Hugging Face q_proj, k_proj, and v_proj tensors. Other layers were mapped directly to their corresponding Llama components.

Numerical validation

The native model and the converted Hugging Face model were evaluated with identical input token IDs.

For the released checkpoint:

• Maximum absolute logit difference: approximately 9.54e-6
• Mean absolute logit difference: approximately 1.14e-6
• Logit cosine similarity: 1.0000000000
• Next-token argmax: identical
• Top-token ordering: identical in the validation test

Autoregressive generation was also tested using the same random seed and sampling configuration.

The native implementation and Hugging Face implementation produced an exact token-for-token match using:

• temperature: 0.5
• top-k: 30
• top-p: 1.0

Hugging Face generation with KV caching was also verified against full-context recomputation.

Sample Generation

PROMPT: Photosynthesis is a process
OUTPUT: Photosynthesis is a process that can release energy from nutrients and improve the ability of plants to digest food. <|endoftext|>

Usage

my_prompt = "Photosynthesis is a process"

inputs = tokenizer(
    my_prompt,
    return_tensors="pt",
    add_special_tokens=False
)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.5,
        top_k=30,
        top_p=1.0,
        use_cache=True,
        pad_token_id=1,
        eos_token_id=3,
    )

generated_text = tokenizer.decode(
    output[0],
    skip_special_tokens=False
)

print("PROMPT:", my_prompt)
print("OUTPUT:", generated_text)