HOLA 340M

This is the main HOLA checkpoint used in the paper.

Model details

• Architecture: Gated DeltaNet with HOLA bounded exact cache
• Scale: 340M parameters
• Training data: SlimPajama-15B
• Tokenizer: Mistral-7B-v0.1 tokenizer, vocab size 32000
• Training context: 2048 tokens
• Cache: w=64, chunk=256, RMSNorm-gamma cache read
• Eviction: betae (HOLA, surprise eviction by beta * ||e||)

Usage

Install the HOLA code package first, then load with Hugging Face Auto classes:

import torch
import fla  # registers gated_deltanet
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "YOUR_ORG/hola-340m"
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    path, torch_dtype=torch.bfloat16, device_map={"": "cuda"}
).eval()

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

For the recency-control model, generation-style evaluations should set:

export FLA_FORCE_NO_CACHE=1

Paper evaluation snapshot

Citation

@article{hola2026,
  title = {HOLA: Hybrid Optimization with a Learned Attention Cache},
  author = {TODO},
  journal = {arXiv preprint},
  year = {2026}
}