tabUM v1.1
A small tabular in-context-learning foundation model: 14.31M parameters, zero-shot classification and regression on any table, native missing-value handling, up to 100 classes and 2,000 features. MIT licensed, built clean-room (no third-party model code).
Code, notebook, and full results: github.com/heldernoid/tabum
What it does
tabUM never trains on your data. fit(X, y) stores your rows; predict(X_test)
runs a single forward pass in which every test row attends to every training row
(in-context learning). Pretraining happened once, on millions of synthetic tasks
sampled from causal graphs, Gaussian processes, and random tree ensembles.
- prediction in seconds, no tuning, no preprocessing pipelines
NaNis a first-class value: missingness is signal, never imputed- calibrated probabilities; full regression distributions (any quantile) in one pass
- up to 100 classes (most tabular foundation models cap at 10)
- optional
finetune()(~15s per-dataset adaptation) andexplain()(feature importances + the exact training rows the prediction attended to)
Usage
pip install git+https://github.com/heldernoid/tabum
from huggingface_hub import snapshot_download
from tabum.model import TabUM
from tabum.inference import TabUMClassifier, TabUMRegressor
model = TabUM.from_pretrained(snapshot_download("helmo/tabum-v1.1"), device="cuda")
clf = TabUMClassifier(model=model, n_ensemble=8).fit(X_train, y_train)
proba = clf.predict_proba(X_test)
reg = TabUMRegressor(model=model, n_ensemble=8).fit(X_train, y_train)
p90 = reg.predict_quantile(X_test, 0.9)
Evaluation
TabArena-v0.1, all 51 datasets, identical protocol for every model (at most 2,000 train / 1,000 test rows, single split, seed 0; competitors run as black-box pip packages on the same splits). Per-dataset numbers are in the GitHub repository.
| model | params | cls accuracy (38) | reg R² (13) |
|---|---|---|---|
| logistic / linear regression (fitted) | 0.8417 | 0.5597 | |
| tabUM zero-shot, 1 pass | 14M | 0.8523 | 0.6284 |
tabUM zero-shot, n_ensemble=8 | 14M | 0.8581 | 0.6519 |
tabUM finetune() + n_ensemble=8 | 14M | 0.8635 | 0.6907 |
| HistGradientBoosting (fitted, default) | 0.8669 | 0.7091 | |
| TabPFN v2 (zero-shot) | ~11M | 0.8768 | 0.7471 |
| TabICL (zero-shot) | ~500M | 0.8801 | n/a |
Strengths and honest limitations:
- beats fitted linear baselines everywhere, zero-shot
- finetuned, it ties default gradient boosting on classification (20/38 wins) and trails it by 1.8 R² points on regression
- 1.3 to 1.7 accuracy points behind TabPFN v2 and TabICL (the state of the art in this class), at 14M parameters and MIT license
- many-class is a differentiator: letter (26 classes) scores 0.79 vs 0.74 for fitted logistic regression; TabPFN-class models cap at 10 classes
- strongest regime: small noisy tables (under ~1,000 rows), where the learned prior beats boosting; on Titanic with 12% missing cells fed raw it beats an imputed+scaled logistic regression pipeline
- context lengths validated to 64k rows (accuracy keeps improving with more context)
Model details
| architecture | 3-stage tabular ICL transformer: Fourier cell embeddings with inducing-point column attention, per-row CLS aggregation, 10-block row transformer (train rows bidirectional, test rows attend train-only), retrieval-attention classification head, bar-distribution regression head |
| parameters | 14,306,051 |
| training | 20,000 steps of continued pretraining from tabUM v1 (50,000 steps), on synthetic tasks only, streamed on the fly (no task seen twice), 50/50 classification/regression, up to 16,384 rows / 2,000 features / 100 classes |
| hardware | one NVIDIA DGX Spark (GB10, 128GB unified memory), ~7h |
| data | 100% synthetic (causal graphs / Gaussian processes / random trees); no real-world data, no OpenML data, and no other model's outputs were used in training |
| license | MIT (weights and code) |
Intended use and limitations
Intended for tabular prediction on datasets from roughly 50 to tens of thousands of rows, up to 2,000 features, up to 100 classes. Not a language model: free-text columns should be dropped or encoded upstream. Regression on large, low-noise datasets is better served by gradient boosting; tabUM's edge is speed-to-first-prediction, small data, many classes, missing values, and built-in explanations.