RWKV7-G1j-13.3B-20260831

This repository provides the RWKV-7 G1j 13.3B checkpoint in the flash-linear-attention (FLA) RWKV7 layout for Transformers-compatible inference.

[!IMPORTANT] This is a base language model, not a safety-aligned instruction-tuned assistant. The included chat template provides a conversational prompt format, but the model may not follow instructions consistently.

About RWKV-7

RWKV-7, also called Goose, is an attention-free recurrent language model. It maintains a constant-size recurrent state instead of an attention KV cache that grows with the preceding sequence. Training remains parallelizable, while recurrent decoding uses constant state size and constant work per generated token with respect to sequence length.

G1j identifies this checkpoint revision. The source checkpoint is available from BlinkDL/rwkv7-g1.

Model details

Run with FLA

Use a CUDA-capable NVIDIA GPU with BF16 support. Install the audited FLA revision with its CUDA dependency extra:

python -m pip install \
  "flash-linear-attention[cuda] @ git+https://github.com/fla-org/flash-linear-attention.git@8e84ed4a6727be082c34a3855c60623fd11411e9" \
  "transformers>=4.50.2"

Import fla before using the Auto classes so that the RWKV7 implementation is registered:

import fla
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedConfig

model_id = "fla-hub/RWKV7-G1j-13.3B-20260831"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    config=PreTrainedConfig(),
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()

messages = [
    {"role": "user", "content": "Explain recurrent language models in one paragraph."}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

new_tokens = output_ids[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Set thinking=True when applying the chat template to leave an open thinking prefix. This changes only the prompt format; it does not turn the base model into an instruction-tuned or safety-aligned assistant.

Tokenizer

This repository bundles tokenization_rwkv7.py, an exact linear-time trie implementation of the RWKV World tokenizer. It reads the self-contained tokenizer.json, preserves canonical token IDs, and avoids the poor scaling of the generic Unigram tokenizer on long repetitive inputs. Loading it requires trust_remote_code=True, as shown above.

Compatibility and validation

The package was checked with FLA commit 8e84ed4a6727be082c34a3855c60623fd11411e9 and Transformers 4.50.2. Its configuration, tokenizer, chat template, BF16 weights, and Transformers model loading were validated locally. All 2,016 model tensors load into RWKV7ForCausalLM without missing, unexpected, or mismatched keys.

CUDA/Triton generation was not executed in the local CPU-only validation environment. Runtime behavior can depend on the GPU, CUDA, PyTorch, Triton, and FLA versions. Cross-check benchmark or evaluation results against the official RWKV implementation before reporting them.

References

• RWKV-7 paper
• RWKV-LM
• Flash Linear Attention
• Source checkpoint repository