PIT-4B — Point-In-Time GPT (Pre-trained, 2019-12)
Point-In-Time (PIT) is a family of GPT-style language models trained on chronologically-ordered monthly snapshots of FineWeb. Each checkpoint captures the state of knowledge available up to a specific month, making them suitable for temporal reasoning and point-in-time analysis tasks.
This is the base (pre-trained only) variant. For instruction-following, use the corresponding PIT-4B-FT checkpoint.
Model details
| Property | Value |
|---|---|
| Snapshot month | 2019-12 |
| Architecture | Decoder-only Transformer (GPT) |
| Layers | 20 |
| Hidden dim | 4096 |
| Attention heads | 32 |
| Vocab size | 50304 |
| Tokenizer | GPT-2 BPE |
| Position encoding | RoPE |
| Normalization | RMSNorm on Q/K + pre-norm |
| Activation | Squared ReLU |
| Weight tying | Yes (input emb ↔ lm_head) |
Requirements
pip install transformers torch safetensors
Quick start
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo_id = "Diamegs/PIT-4B-201912"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True, # required for custom architecture
torch_dtype=torch.bfloat16,
)
model = model.cuda()
model.eval()
Text generation
prompt = "In 2019, the global economy"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.8,
top_p=0.95,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))
Temporal reasoning example
Because this model was trained on data up to 2019-12, it reflects the world as it was known at that point. You can use this for point-in-time analysis:
# What does the model "know" about events before its cutoff?
prompt = "The most important AI developments in early 2019 were"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
max_new_tokens=150,
do_sample=True,
temperature=0.7,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))
Weights format
Weights are stored in safetensors format
(model.safetensors) — memory-mapped, fast to load, and safe (no arbitrary code execution).
Limitations
- Knowledge is limited to web text available up to 2019-12.
- No RLHF or safety fine-tuning has been applied (base model).
- The model may reproduce biases present in FineWeb training data.
- Not suitable for safety-critical applications without further alignment.
Citation
If you use these models in your research, please cite our paper:
@techreport{kelly2026pit,
title = {Scaling Point-in-Time Language Models},
author = {Kelly, Bryan T. and Malamud, Semyon and Schwab, Johannes and Xu, Teng Andrea},
institution = {Swiss Finance Institute},
type = {Research Paper},
number = {26-37},
year = {2026},
month = apr,
doi = {10.2139/ssrn.6681860},
url = {https://ssrn.com/abstract=6681860}
}