Tabby-Pretrain
The pretrained PatchTST backbone behind Tabby. It is a encoder only patch transformer trained with a masked-patch objective and a 99-quantile pinball loss, so a single forward pass returns a full predictive distribution.
This repository holds the frozen backbone only — use it for zero-shot forecasting, or as a base to adapt.
Looking for the full model? The post-trained model, backbone and prompt merged into one checkpoint that runs on its own, is published as Tabby. The prompt module on its own is Tabby-Prompt.
Architecture
| Parameters | 145.84 M (253 tensors) |
| Layers | 20 |
d_model | 768 |
| Attention heads | 12 (head dim 64) |
| MLP hidden | 3072 (4× d_model) |
| Patch size | 16 |
| Context window | 8192 steps (512 patches) |
| Quantile levels | 99 (0.01 … 0.99) |
| Dropout | 0.1 |
| Additional pretraining | 165,000 steps after a 10,000-step initialization checkpoint |
| Precision | released in fp32; trained in bf16 |
The quantile head is monotone by construction: it predicts a base level plus cumulative softplus increments, so the 99 levels can never cross.
Files
| File | What it is |
|---|---|
model.safetensors | backbone weights, 583 MB, fp32 |
config.json | architecture config |
training_args.json | the recorded pretraining hyperparameters (dataset paths redacted) |
The published weights are bit-for-bit the model state dict of the original
training checkpoint. Only the optimizer state was dropped, which is what takes
the file from 1750 MB down to 583 MB. If you need to resume pretraining rather
than run inference, the optimizer state is not in this repository.
Usage
The loader in the Tabby code recognises a config.json + model.safetensors
directory and loads it with no key renaming:
from huggingface_hub import snapshot_download
from tabby_prompt.model import load_patchtstfm_backbone
path = snapshot_download("paris-noah/Tabby-Pretrain")
backbone, cfg, step = load_patchtstfm_backbone(path, device="cuda", code_dir=PRETRAIN_CODE)
The loader reads a cumulative checkpoint counter from model.safetensors
metadata. That counter includes the 10,000-step initialization checkpoint;
the training duration reported here is the subsequent 165,000 steps.
To use it zero-shot through the same predict path Tabby uses, wrap it with
prompt_len=0:
from tabby_prompt.model import PatchTSTFMPromptCFG, PromptedPatchTSTFM
cfg = PatchTSTFMPromptCFG(prompt_len=0, context_aware=False,
prediction_length=96, context_length=8096)
model = PromptedPatchTSTFM(backbone, cfg).to("cuda").eval()
out = model(context=x, prediction_length=96) # out["quantile_preds"]: (B, 99, H)
Zero-shot reference
Measured through the Tabby evaluation harness at context_length=8096,
each variate forecast independently:
| Benchmark | MASE | CRPS |
|---|---|---|
GIFT-Eval, 97 configs, max(H, 128) inference | 0.7040 | 0.4827 |
These are Seasonal-Naive normalised geometric means from the supplied GIFT-Eval evaluation CSV. Updated TIME window-level outputs are being submitted for official leaderboard aggregation; no TIME aggregate is claimed here.
Training
Initialized from a 10,000-step checkpoint, then trained for 165,000 steps in bf16 on a mixture of real and synthetic series:
- real series from BLAST (ratio 0.3) and a GIFT-Eval pretraining pool (ratio 0.3), both domain-balanced;
- synthetic series from a Chronos KernelSynth corpus (ratio 0.3), plus CauKer-V2 (ratio 0.1).
Objective: masked patch modelling with mask_ratio=0.4 in contiguous blocks of 8
patches, plus terminal masking of 0–2 patches, scored with a 99-quantile pinball
loss. Intermediate supervision is applied at layer exits [0, 5, 10, 15, 20]
(lambda_ds=0.5, lambda_fm=0.1).
Optimizer: AdamW, β=(0.9, 0.95), weight decay 0.1, gradient clip 1.0, WSD schedule with 10,000 warmup and 20,000 decay steps, peak and floor LR both 1e-5, effective batch 1200 (200 × 6 accumulation steps).
Full values are in training_args.json.
License
CC BY-NC 4.0 — non-commercial use only.