Avey-1

⚠️ RESEARCH PREVIEW - NOT FOR PRODUCTION USE

This is a research preview for the Avey-1 language model, a recurrence- and attention-free autoregressive language model that decouples context width from total sequence length, introduced in the paper Don't Pay Attention. This version of the model is not expected to be able to generate high-quality text.

Model Description

Architecture

Avey-1 comprises two principal components, a ranker and a neural processor. The ranker slices each input sequence into splits of consecutive tokens and selects the top K most relevant splits for each current split being processed by the neural processor. The neural processor consists of three core units, the enricher, contextualizer, and fuser. The enricher enhances the quality of token embeddings by expanding their learnable features using a position-wise neural network. The contextualizer is an embedding-wise neural network with dynamic parameterization, enabling interactions between relevant tokens across the current and top K splits. Lastly, the fuser learns a function that integrates the contextualized features produced by the contextualizer with some uncontextualized features bypassed by a partial-embedding bypassing mechanism.

For further details, please refer to the paper Don't Pay Attention, and the code at https://github.avey.ai/avey-dpa.

Tokenizer

This model uses the p50k_base tokenizer, converted to a HuggingFace transformers compatible tokenizer using this script.

Parameters

• Vocabulary size: 50280
• Split Size: 128
• Top K: 3
• Embedding Dimension: 2048
• Number of Layers: 48
• Expansion Factor: 4
• Tail proportion: 0.5

Training

• Pre-training: 1 epoch on 100B tokens (FineWeb sample-100BT) with S=64 and K=7
• Fine-tuning: 1 epoch on 1B tokens (FineWeb sample-10BT) with S=128 and K=3
• Infrastructure: 128× NVIDIA H200 GPUs

For full details and training scripts, see the GitHub repository and the research paper.

Note: With only 100B tokens of pre-training, the Avey-1 research preview model is likely under-trained and will produce much lower‐quality text compared to production‐scale models.

Intended Uses & Limitations

Intended for:

• Research experiments
• Architecture benchmarking

Not intended for:

• Production deployments
• downstream applications

Limitations:

• Undertrained: outputs can be incoherent or low‐quality
• Unoptimized implementation: slower inference than other optimized models
• Base model: best results will require further pretraining and downstream fine-tuning

Quick Start: Text Generation

Requirements:

• Python 3.8+
• PyTorch
• Transformers

This code should be able to run on GPUs with 8GB of VRAM or more. It will download the model checkpoints, which should take about ~6GB of storage space.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

prompt = "Avey is a new machine-learning language model architecture that"

model_id = "avey-ai/avey1-1.5B-base-preview-100BT"

# Select device
if torch.cuda.is_available():
    device = "cuda"
elif torch.backends.mps.is_available():
    device = "mps"
else:
    device = "cpu"

# Load model (use bf16 on supported GPUs for faster inference)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32
).to(device)

# required for linear time generation
model.set_to_generation_mode()

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Prepare input
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Stream generation
streamer = TextStreamer(tokenizer)
_ = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.8,
    min_p=0.1,
    repetition_penalty=1.1,
    streamer=streamer,
)

⚠️ This model is provided for research purposes only. It is not optimized for production and will generate sub-optimal or unpredictable outputs.