Llama 1B with LongCat n-gram embeddings (25%), 6B tokens

A ~1B-parameter Llama 3-style decoder that spends about 25% of its parameter budget on hashed n-gram embedding tables, following the LongCat n-gram embedding module. To keep the total near 1B, it has 23 transformer layers instead of the baseline's 32. Everything else matches the baseline, including the 6B FineWeb training tokens.

This is a base model. It is not instruction-tuned or safety-tuned.

Results

Training

Zero-shot benchmarks

Scores from lm-eval on each task's full split. Shared-9 is the unweighted mean of the nine tasks.

Few-NERD (LoRA fine-tuned)

Usage

The architecture class ships with the checkpoint, so load it with trust_remote_code=True. Tested with transformers==5.8.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Mercity/pretrain-longcat-ngram-25pct"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Cached generation works: the model rebuilds the n-gram context from the full token history at each decoding step.

To score text instead of generating it, pass labels and read the loss:

batch = tokenizer("FineWeb is a large web-text dataset.", return_tensors="pt").to(model.device)
with torch.no_grad():
    loss = model(**batch, labels=batch["input_ids"]).loss
print(f"loss={loss.item():.3f}  ppl={loss.exp().item():.1f}")

When quantizing, keep the n-gram tables in higher precision. They are embedding lookups, not ordinary weight matrices.

Model details

Training

Related checkpoints

Limitations

Trained on 6B English web tokens only, a small budget for a 1B model. Benchmark scores are single-seed. The model will repeat or make up facts and has had no alignment training.