ZGCM-1-7B-Pretrain-Curriculum
Checkpoint after the shuffled second pass of the curriculum data and before the subsequent new-data pretraining stage. The historical 1T label refers to the curriculum data naming, not the cumulative training-token count.
| Property | Value |
|---|---|
| Checkpoint step within its run | 138000 |
| Training context | 16,384 tokens |
| Training tokens (pretraining cumulative) | approximately 1634.87B |
| Precision | BF16 |
| RoPE base | 5,000,000 |
| Stage completed | Yes |
This is a base language-model checkpoint. It has not undergone instruction tuning. Use plain text inputs; the included tokenizer assets retain the original token IDs.
Loading
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "zgcagi/ZGCM-1-7B-Pretrain-Curriculum"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True, torch_dtype=torch.bfloat16,
device_map="auto", attn_implementation="eager",
)
inputs = tokenizer("The purpose of scientific research is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The repository includes the custom ZGCM Transformers implementation. The loading example uses its uncached reference generation path. Conversion and short forward/ generation checks used PyTorch 2.7 and Transformers 4.57.0.
Conversion validation
Exported with export_megatron_zgcm_to_hf.py. All 259 source model tensors round-trip
exactly through the 382 exported tensors, preserving BF16 values, grouped Q/G/K/V
layout, SwiGLU weights, and normalization parameters. Every exported shard is
listed in model.safetensors.index.json and SHA256SUMS.
Local HF loading had no missing or unexpected tensors. A 160-token forward pass and short greedy generation passed. These checks do not constitute a native Megatron-versus-HF logits comparison or long-context benchmark validation.
See checkpoint_info.json for the precise checkpoint identity and validation scope.