kairos

KairosFM is a hybrid MoE diffusion language model combining DeltaNet (linear attention), Sliding Window Attention, and Attention Residuals (AttnRes), trained on text, image, video, audio, lidar, and control (state/action) modalities through a shared multimodal conv-byte tokenizer. See github.com/fabienfrfr/Kairos for the full architecture writeup.

This checkpoint

Note: this repo currently tracks best-training-loss only (checkpoints/best.pt) — no held-out validation split is evaluated during training yet.

Files

• checkpoints/ — best.pt (lowest avg training loss) + periodic step_*.pt
• tensorboard/ — events.out.tfevents.*, viewable in the Hub's Training Metrics tab
• config.json, model.safetensors — native HF format, loadable via trust_remote_code

Usage

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("ffurfaro/kairos", trust_remote_code=True)

Requires the kairos package importable (custom architecture, not upstream transformers) — install from github.com/fabienfrfr/Kairos first, or add it to PYTHONPATH. Alternatively, skip Auto* and import the class directly:

from kairos.modeling import KairosDiffusionLLM

model = KairosDiffusionLLM.from_pretrained("ffurfaro/kairos")

Limitations

Experimental, low-compute-budget training run — expect uneven quality across modalities (multimodal data is a small fraction of total training). Not evaluated for safety-critical use.

Citation

@misc{kairos,
  title  = {KairosFM: less parameters, more signal — a multimodal MoE diffusion model for edge AI},
  author = {Fabien Furfaro},
  url    = {https://github.com/fabienfrfr/Kairos}
}