CyFA-800M-30B

Cyclic Flow Attention (CyFA), an 800M-parameter base language model trained on 30B tokens from SlimPajama.

Paper | Code

Usage

Install CyclicFlowAttention, then load the model:

import torch
import cyclic_flow_attention
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "cyxxxxxxxxxx/cyfa-800M-30B"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint, dtype=torch.bfloat16,
).cuda().eval()

inputs = tokenizer("The library opens at", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=64, do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))