Chris-GPT-2 124M GGUF

GGUF export of Chris-GPT-2 124M, a GPT-2-style causal language model trained from scratch on approximately 10 billion tokens from FineWeb-Edu.

The original model was independently pretrained from randomly initialized weights and later converted to Hugging Face Transformers and GGUF formats.

This repository contains the GGUF version of the model for experimentation with lightweight and custom inference runtimes.

Original Model

The canonical Hugging Face Transformers version is available at:

https://huggingface.co/christianrss/chris-gpt-2-124m

The original training implementation is available at:

https://github.com/christianrss/chris-gpt-2

Model Details

The model uses a padded vocabulary size of 50,304 internally while retaining the original GPT-2 BPE tokenizer with 50,257 tokens.

Available GGUF

Additional quantized versions may be added in the future.

Training

Chris-GPT-2 was pretrained from randomly initialized weights rather than fine-tuned from an existing GPT-2 checkpoint.

The training target was approximately 10 billion tokens:

524,288 tokens/step × 19,073 steps
≈ 10,000,000,000 tokens

Training was performed using:

• 4× NVIDIA A100 PCIe 40 GB GPUs
• Distributed Data Parallel (DDP)
• AdamW optimizer
• GPT-2 byte-level BPE tokenization
• 1,024-token context length
• FineWeb-Edu training data

The final checkpoint was produced at step 19,072.

Final validation loss:

3.0724804401397705

Dataset

The model was trained on approximately 10 billion tokens from FineWeb-Edu.

FineWeb-Edu is an educationally filtered subset of FineWeb designed for language-model pretraining research.

Conversion Pipeline

The original model was trained using a custom PyTorch GPT-2 implementation.

The complete conversion pipeline was:

Chris-GPT training
        │
        ▼
model_19072.pt
        │
        │ custom checkpoint conversion
        ▼
Hugging Face GPT2LMHeadModel
        │
        ▼
model.safetensors
        │
        │ GGUF conversion
        ▼
Chris-GPT-2-124M-F16.gguf

The Hugging Face intermediate model was validated by loading it with:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "christianrss/chris-gpt-2-124m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

and successfully performing autoregressive text generation.

Hugging Face Conversion

The original Chris-GPT implementation uses torch.nn.Linear for GPT-2 projection layers, whereas Hugging Face GPT-2 uses its Conv1D representation.

The following weights were transposed during the Hugging Face conversion:

attn.c_attn.weight
attn.c_proj.weight
mlp.c_fc.weight
mlp.c_proj.weight

The conversion process performs tensor-level verification after mapping the original checkpoint parameters.

The resulting Hugging Face model is reloaded with GPT2LMHeadModel.from_pretrained() and validated with a deterministic forward pass before GGUF conversion.

Example Generation

The Hugging Face version of the converted checkpoint was tested with the following prompt:

The future of artificial intelligence is

Using:

temperature = 0.8
top_k       = 50
top_p       = 0.95

the model produced:

The future of artificial intelligence is a critical factor in the global economy. It’s also an important factor in our society’s ability to cope with global warming.
What are the future of artificial intelligence?
Artificial intelligence has the potential to become an important component

This is an actual sampled generation from the trained model and was not manually curated.

GGUF Format

GGUF is a binary format designed for efficient storage and loading of model tensors and metadata.

Unlike the original PyTorch checkpoint, the GGUF file packages the information required by compatible inference runtimes into a portable model artifact.

This repository currently provides an F16 GGUF export.

Chris-GPT-2-124M-F16.gguf

F16 preserves substantially more numerical precision than aggressive low-bit quantization and serves as the reference GGUF representation of Chris-GPT-2.

Chris Llama

The GGUF export is also being used as a reference model for development of Chris Llama, an experimental low-level LLM inference runtime.

Repository:

https://github.com/christianrss/chris-llama

Chris Llama explores topics including:

• GGUF parsing
• tensor loading
• GPT-2 inference
• KV caching
• quantization
• CPU inference
• heterogeneous compute
• AdaptiveCpp
• SYCL
• GPU acceleration

The objective is to study the implementation of LLM inference infrastructure at a lower level rather than relying exclusively on existing high-level runtimes.

Using with Transformers

Hugging Face Transformers supports loading GGUF checkpoints for supported architectures.

Install the required packages:

pip install transformers gguf torch

Then:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "christianrss/chris-gpt-2-124m-GGUF"
filename = "Chris-GPT-2-124M-F16.gguf"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    gguf_file=filename,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    gguf_file=filename,
)

prompt = "The future of artificial intelligence is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
)

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True,
    )
)

When loaded through Transformers, GGUF weights may be converted back into a PyTorch-compatible representation for execution.

For normal Transformers usage, the canonical SafeTensors repository is recommended:

https://huggingface.co/christianrss/chris-gpt-2-124m

Evaluation

Validation was performed throughout pretraining.

Final validation loss:

3.07248

The training pipeline also included HellaSwag evaluation during development.

Chris-GPT-2 should not be interpreted as competitive with modern large language models.

The purpose of the experiment is primarily to study and reproduce the complete lifecycle of a GPT-2-scale Transformer:

dataset
   ↓
tokenization
   ↓
Transformer implementation
   ↓
distributed pretraining
   ↓
validation
   ↓
checkpointing
   ↓
evaluation
   ↓
inference
   ↓
Hugging Face conversion
   ↓
SafeTensors
   ↓
GGUF conversion
   ↓
custom inference runtime

Limitations

Chris-GPT-2 is a relatively small base autoregressive language model.

It has not undergone:

• instruction tuning
• reinforcement learning from human feedback (RLHF)
• preference optimization
• safety fine-tuning
• conversational alignment

The model may generate:

• incorrect information
• repetitive text
• incoherent continuations
• biased content
• undesirable content

It should not be used as an authoritative source of factual information.

The model was developed primarily as a research and engineering experiment, rather than as a production conversational assistant.

Purpose of this Repository

This repository exists specifically to distribute the GGUF representation of Chris-GPT-2.

The project is organized into separate artifacts:

Transformers / SafeTensors

christianrss/chris-gpt-2-124m

Canonical Hugging Face representation for use with Transformers.

GGUF

christianrss/chris-gpt-2-124m-GGUF

Portable GGUF representation for inference and runtime experimentation.

Training Code

christianrss/chris-gpt-2

Original GPT-2 training implementation and experiment code.

Experimental Runtime

christianrss/chris-llama

Low-level inference runtime and GGUF experiments.

Related Projects

Chris-GPT-2

GPT-2 124M pretraining implementation:

https://github.com/christianrss/chris-gpt-2

Chris Torch

Experimental machine-learning framework with custom autograd, neural-network components, optimizers, and native compute backends:

https://github.com/christianrss/chris-torch

Chris Llama

Experimental low-level LLM inference runtime:

https://github.com/christianrss/chris-llama

Research

The pretraining experiment is documented in:

A Reproducible 10-Billion-Token Pretraining Run of GPT-2 124M

ResearchGate:

https://www.researchgate.net/publication/412096883_A_Reproducible_10-Billion-Token_Pretraining_Run_of_GPT-2_124M

The report discusses:

• training methodology
• FineWeb-Edu
• training dynamics
• validation
• HellaSwag evaluation
• generation behavior
• compute infrastructure
• training cost
• reproducibility

References

• Alec Radford et al. Language Models are Unsupervised Multitask Learners. OpenAI, 2019.
• Ashish Vaswani et al. Attention Is All You Need. 2017.
• Hugging Face FineWeb / FineWeb-Edu.
• Hugging Face Transformers.
• GGUF / GGML.
• llama.cpp.
• Andrej Karpathy, Let's reproduce GPT-2 (124M).

Author

Christian Rafael de Souza Silva

Independent Researcher

GitHub:

https://github.com/christianrss